Preempt sensu-plugin `occurrences` and `refresh` functionality with a simple / lightweight filter

occurences.md

UPDATE (2017-10-20): it has recently been brought to my attention that this gist has been circulated amongst Sensu users as a sort of best practice or "guide". This gist was never meant to be adopted as a long-term solution, but rather an alternative/temporary solution while many changes regarding Sensu event filtering were "in-flight". Many more words regarding changes to event filtering have been captured on the Sensu blog since the writing of this gist: https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3. The TL;DR here is: use the new built-in occurrence filtering that has been available in Sensu Core since version 0.26.0 (release notes).

Hi friends.

There has been a lot of confusion over the past few months about how the Sensu check definition attributes occurrences and refresh actually work. The attributes occurrences and refresh are not (currently) a part of the Sensu check definition specification — they are actually just custom check definition attributes which are supported by the sensu-plugin gem's Handler class. Recently, the documentation around these attributes has moved from the Checks reference documentation to the Plugins reference documentation (see: plugin definition specification), in an attempt to make it more clear how/where these attributes are used.

At Heavy Water, we've always been cautious about recommending the use of these attributes as they rely on handler plugins that use the sensu-plugin gem, which typically results in confusion when a team eventually (yet unknowingly) starts using a new handler that doesn't support these attributes (e.g. if a handler isn't written in Ruby, and/or if a Ruby-based handler isn't using the sensu-plugin gem). Inevitably someone ends up flipping a table and saying something like this:

"I'm setting occurrences and refresh but Sensu is not working!".

We appreciate how frustrating this is (even though it is technically incorrect), so we are taking several steps to improve it:

1. We have added support for Filter attribute eval tokens in Sensu version 0.23, significantly improving the usefulness of filters.
2. We have improved the documentation, clearly separating the definition of the occurrences and refresh attributes from the check definition specification (occurrences and refresh have been moved to the plugins reference documentation), and explaining more clearly that anything that's not in the specification is not directly supported by Sensu, and simply a custom definition attribute.
3. We have opened a discussion about the larger issue surrounding first-class support for "subduing" handling (and/or check execution) in Sensu proper, which changes we are targeting for a version 0.25 release (~June 2016).

Example filter

In the interim, the following example filter definition may be used as a globally applicable replacement for the sensu-plugin gem handling of occurences and refresh.

{
  "filters": {
    "refresh": {
      "negate": true,
      "attributes": {
        "occurrences": "eval: value != :::check.occurrences::: && value % :::check.refresh::: != 0"
      }
    }
  }
}

To use this filter, create a filter definition at /etc/sensu/conf.g/refresh.json, and add the refresh filter to your handler definitions; e.g.:

{
  "handlers": {
    "example_handler": {
      "type": "pipe",
      "command": "notify_errrbody.rb",
      "filters": [
        "refresh"
      ]
    }
  }
}

NOTE: this is not a backwards-compatible replacement for the refresh functionality provided by the sensu-plugin gem, which expects refresh to be a value in seconds, and does some math to determine when to "refresh" the handling. In this example filter, refresh values should be provided in terms of "occurences"; e.g. if a check has an interval of 30 (seconds), and you want to refresh handling of events from said check every hour, you would need to set "refresh": 120 instead of "refresh": 3600. Technically, you could do a little more math in this filter to make it backwards-compatible, but it starts to get a little ugly and it defeats the purpose of filters, which are intended to be used for lightweight attribute comparison.

Please leave a comment if this has helped you, or if you have any questions.

#monitoringlove

Please use the built-in occurrence filter 👍