Sensu Remediate Example

gistfile1.txt

1. Download the plugin from: https://github.com/sensu/sensu-community-plugins/blob/master/handlers/remediation/sensu.rb 
2. Create the handler definition:
{                                                    
 "handlers": {                                       
   "remediator": {                                   
     "type": "pipe",                                 
     "command": "/etc/sensu/handlers/remediator.rb"  
    }                                                
  }                                                  
}      

3. Create check definition:
{                                                                                                  
  "checks": {                                                                                      
    "seyren_check": {                                                                              
      "command": "/etc/sensu/plugins/check-procs.rb -p PROC",                                    
      "interval": 10,                                                                              
      "subscribers": ["SUB"],                                                                 
      "handlers": ["remediator","pagerduty"],                                                      
      "occurrences": 1,                                                                            
      "refresh": 10,                                                                               
      "remediation": {                                                                             
        "light_remediation": {                                                                     
          "occurrences": [1, 2],                                                                   
          "severities": [1]                                                                        
        },                                                                                         
        "medium_remediation": {                                                                    
          "occurrences": ["3-10"],                                                                 
          "severities": [1]                                                                        
        },                                                                                         
        "heavy_remediation": {                                                                     
          "occurrences": ["1+"],                                                                   
          "severities": [2]                                                                        
        }                                                                                          
      }                                                                                            
    },                                                                                             
    "light_remediation": {                                                                         
      "command": "touch /tmp/test",            
      "subscribers": [],                                                                           
      "handlers": ["pagerduty"],                                                                   
      "pager_team": "testing",                                                                     
      "interval": 10,                                                                              
      "publish": false                                                                             
    },                                                                                             
    "medium_remediation": {                                                                        
      "command": "touch /tmp/test",            
      "subscribers": [],                                                                           
      "handlers": ["pagerduty"],                                                                   
      "pager_team": "testing",                                                                     
      "interval": 10,                                                                              
      "publish": false                                                                             
    },                                                                                             
    "heavy_remediation": {                                                                         
      "command": "touch /tmp/test",            
      "subscribers": [],                                                                           
      "handlers": ["pagerduty"],                                                                   
      "pager_team": "testing",                                                                     
      "interval": 10,                                                                              
      "publish": false                                                                             
    }                                                                                              
  }                                                                                                
}   

4. I noticed in the Sensu-API log that remediator was POSTing a check to the API with the subscribers being the hostname of the affected server. Sample log message:

{"timestamp":"2014-10-06T18:46:16.656712+0000","level":"info","message":"POST /request","remote_address":"127.0.0.1","user_agent":"Ruby","request_method":"POS
T","request_uri":"/request","request_body":"{\"check\":\"heavy_remediation\",\"subscribers\":[\"hostname.example.com\"]}"}  

5. I changed my test client to subscribe to that queue and success. Everything works now:
{                                                    
  "client":{                                         
    "subscriptions":[                                
      "test",                                    
      "hostname.example.com"              
    ],                                               
    "name":"hostname.example.com",        
    "address":"555.555.555.555"                       
  }                                                  
}                                                    

hi, adding the hostname to the subscription will clutter the dashboard. Every client will be a subscription in the frontend. Is there a better approach?

Agreed with above comment! this shall create unnecessary 'subscription' which occupies Redis backend and also clutters the subscriptions, so also additional update to the client.json ifile !

What could be other possible way out! I wish Sensu API should have a way to push the checks for only failing nodes...

While not ideal, I believe that by subscribing to the hostname, this is the only way remediation will actually work.

+1 anything new on this?

To avoid cluttering your subscriptions with hostnames, you could put the following line in your seyren_check definition, right after the remediation value:

"trigger_on": ["SUB"]

This will trigger the remediation commands on all clients that subscribe to SUB

Obviously, this solution will only work properly if one of the following conditions is met:

• your remediation command is non-destructive
• your remediation command has some error-checking built in and only runs if the error condition is met
• you only have one server subscribed

Otherwise your remediation command may fail or interrupt services or do other bad things.