sensu-alerts.md

Alerts in Sensu Go

1. Setup
2. Configure Sensu
3. Create a pipeline to send alerts to Slack
4. Create a check to monitor Nginx
5. Create a check to monitor CPU usage

1. Setup

Virtual Box

Download VirtualBox

Vagrant

Download Vagrant

Skunkworks

Download the Sensu Skunkworks repository.

# Download the virtual machine
cd Documents && git clone [email protected]:sensu/skunkworks.git

Install the Vagrant virtual machine running the CentOS operating system and Sensu Core Edition:

# Install the virtual machine
cd skunkworks/standalone && vagrant up

To see what it's doing, check out https://github.com/sensu/skunkworks/blob/master/standalone/sensu2-centos.sh.

Note: You can use vagrant destroy then vagrant up to erase and restart the virtual machine.

Get a Slack webhook URL

If you're already an admin of a Slack, visit https://YOUR WORKSPACE NAME HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration, choose a channel, and save the settings. (If you're not yet a Slack admin, start here to create a new workspace.) After saving, you'll see your webhook URL under Integration Settings.

Access the virtual machine

Once the install process has completed, we can access the virtual machine.

# Access the virtual machine
vagrant ssh

You should see your command prompt change to indicate that you're accessing the virtual machine.

Note: To exit out of the virtual machine, use CTRL+D.

2. Configure Sensu

Sign in to the dashboard

You should see a line in your terminal that starts with Access the dashboard at. Copy and paste that URL into your browser and sign in as the default admin user (username:admin, password:P@ssw0rd!).

Configure and start the Sensu agent

Use Nano to open the Sensu agent configuration file and set the backend URL and add a subscription.

# Open the agent configuration file
sudo nano /etc/sensu/agent.yml

Look for these lines:

#subscriptions: ""
#backend-url:
#  - "ws://127.0.0.1:8081"

Remove the # and add a subscription, so it looks like:

subscriptions: "testing"
backend-url:
  - "ws://127.0.0.1:8081"

To save and exit nano, CTRL+X then Y then ENTER.

Now we'll start the Sensu agent:

# Start the Sensu agent
sudo systemctl start sensu-agent

Configure Sensuctl

sensuctl configure

Enter the following information when prompted:

? Sensu Backend URL: http://127.0.0.1:8080
? Username: admin
? Password: P@ssw0rd!
? Namespace: default
? Preferred output format:
❯ none
  json
  wrapped-json

3. Create a pipeline to send alerts to Slack

To get started, let's create a pipeline to send alerts to Slack. We've already installed this Sensu Slack handler during setup. We can test it using:

# Access help view for Sensu Slack handler
slack-handler -h

Let's create a Slack handler. Copy the following into a text editor and swap in your webhook URL.

sensuctl handler create slack \
--type pipe \
--command 'slack-handler \
  --webhook-url https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX \
  --channel monitoring' \
--filters is_incident,not_silenced

Verify it using:

# Do I have any pipelines set up?
sensuctl handler list

4. Create a check to monitor Nginx

Nginx is a webserver that we installed and started during setup. You can make sure it's running using:

curl -I http://localhost:80

First off, we'll create a check to monitor Nginx.

sensuctl check create check-nginx \
--command 'http_check.sh http://localhost:80' \
--interval 20 \
--subscriptions testing \
--handlers slack

See the events created by this check:

# What events is Sensu receiving?
sensuctl event list

Now we'll stop Nginx so that we receive an alert from Sensu.

sudo systemctl stop nginx

You should see an alert in Slack. Start Nginx to resolve the alert.

sudo systemctl start nginx

Note: Due to a bug in the way Sensu works right now, you won't see a resolution notification in Slack.

5. Create a check to monitor CPU usage on the virtual machine

As part of setup, we installed a check to monitor CPU usage. You can test it using:

# Check CPU usage
check-cpu.sh

To create a check that monitors CPU usage:

sensuctl check create check-cpu \
--command 'check-cpu.sh -w 75 -c 90' \
--interval 20 \
--subscriptions testing \
--handlers slack

And verify with:

# What events is Sensu receiving?
sensuctl event list

To test this check, we'll use a program called Stress that we installed during setup.

# Stress my CPU
stress -c 1

We should get an alert in Slack while Stress is running. Quit stress with CTRL+C to resolve the alert.