goss implementation

crontab

*/5 * * * * cd /opt/healthz && /bin/echo "`date +'\%Y-\%m-\%dT\%H:\%M:\%SZ'` `/usr/local/bin/goss validate -f nagios_verbose`" >> healthz.log

goss.yaml

package:
  apache2:
    installed: true
service:
  apache2:
    enabled: true
    running: true
  tomcat8:
    enabled: true
    running: true
  ufw:
    enabled: true
    running: true
command:
  sudo ufw status verbose:
    # required attributes
    exit-status: 0
    # optional attributes
    stdout:
    - /22 +ALLOW IN/
    - /8080 +ALLOW IN/
    - /8005 +DENY IN/
file:
  /etc/fstab:
    exists: true
    contains: 
    - /tmpfs +\/run\/shm +tmpfs +defaults,noexec,nosuid +0 +0/
  /etc/logrotate.d/tomcat8:
    exists: true
    contains: 
    - size 5M
    - rotate 14
  /etc/logrotate.d/apache2:
    exists: true
    contains: 
    - size 5M
    - rotate 14
  /usr/share/tomcat8/bin/setenv.sh:
    exists: true
    contains:
    - -Xms256m 
    - -Xmx1024m
  /opt/aws-scripts-mon/mon-put-instance-data.pl:
    exists: true

runtests.sh

#!/bin/bash

#Put this in your packer to install and run gossfile on packer build.


# Install goss for testing
# See https://github.com/aelsabbahy/goss/releases for release versions
sudo curl -L https://github.com/aelsabbahy/goss/releases/download/v0.3.2/goss-linux-amd64 -o /usr/local/bin/goss
sudo chmod +rx /usr/local/bin/goss

sudo mkdir /opt/healthz
sudo chown ubuntu:ubuntu /opt/healthz
cp /tmp/files/goss.yaml /opt/healthz

cd /opt/healthz

# Run tests
goss validate

userdata.sh

cd /opt/healthz

# let the elb monitor health
nohup goss serve -f nagios_verbose 2> /dev/null &

goss.yaml file is pushed to the /tmp directory by packer (also seperate)

I install and run goss during the packer build to validate the build worked

The userdata.sh creates a background process to serve the goss healthcheck to the elb. (:8080/healthz is default)
You will need to open ELB security groups and point a HTTP check to :8080/healthz

crontab is used to create a logfile of the goss tests (I send them to cloudwatch logs separately) So when it falls over you can check why.