Skip to content

Instantly share code, notes, and snippets.

@bdha

bdha/gist:8628477

Last active October 13, 2016 18:06
Show Gist options
  • Save bdha/8628477 to your computer and use it in GitHub Desktop.
Save bdha/8628477 to your computer and use it in GitHub Desktop.
Download ZIP
Set up Graphite and Panoptimon on SmartOS
Raw
gistfile1.md

Install Graphite

pkgin up
pkgin in py27-graphite-web ap22-py27-wsgi

If pkgin says the package cannot be found, do this:

wget http://pkgsrc.joyent.com/packages/SmartOS/2013Q3/x86_64/All/py27-graphite-web-0.9.10nb2.tgz
pkg_add py27-graphite-web-0.9.10nb2.tgz
pkgin in ap22-py27-wsgi

Configure Graphite

Update /opt/local/etc/graphite/carbon.conf

LOCAL_DATA_DIR = /var/graphite/whisper/
LOG_DIR=/var/graphite/log
PID_DIR=/var/graphite/log
USER = graphite

Update /opt/local/etc/graphite/graphite.wsgi

Change sys.path.append to /opt/local/graphite/webapp

cd /opt/local/lib/python2.7/site-packages/graphite
cp local_settings.py.example local_settings.py

Edit local_settings.py.

Set the following values:

GRAPHITE_ROOT = '/opt/local'
CONF_DIR = '/opt/local/etc/graphite'
STORAGE_DIR = '/var/graphite'
CONTENT_DIR = '/opt/local/graphite/webapp/content'

DATABASES = {
    'default': {
        'NAME': '/var/graphite/db/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

ALLOWED_HOSTS = ["10.200.70.10", "graphite.engine.sourcefire.com"]

ALLOWED_HOSTS should contain your IP and FQDN. Otherwise Django breaks.

touch /var/graphite/index ; chown www /var/graphite/index
mkdir -p /var/graphite/db
chown www:graphite /var/graphite/db

Edit /opt/local/lib/python2.7/site-packages/graphite/app_settings.py.

Set SECRET_KEY to something unique.

Set up the Graphite database.

python manage.py syncdb
...
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'):
Email address: [email protected]
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
SAXReaderNotAvailable: expat not supported

Fix db perms:

chown -R www:graphite /var/graphite/db

Set up Apache

Include the vhosts file:

perl -pi -e 's:^#Include etc/httpd/httpd-vhosts.conf:Include etc/httpd/httpd-vhosts.conf:' /opt/local/etc/httpd/httpd.conf

Set up the log dir:

mkdir -p /var/graphite/log/webapp
chown www:www /var/graphite/log/webapp

Edit /opt/local/etc/httpd/httpd-vhosts.conf

#
NameVirtualHost *:80

# You need to manually edit this file to fit your needs.
# This configuration assumes the default installation prefix
# of /opt/graphite/, if you installed graphite somewhere else
# you will need to change all the occurances of /opt/graphite/
# in this file to your chosen install location.

<IfModule !wsgi_module.c>
    LoadModule wsgi_module lib/httpd/mod_wsgi.so
</IfModule>

# XXX You need to set this up!
# Read http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGISocketPrefix
WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
        ServerName YOURHOST
        DocumentRoot "/opt/local/graphite/webapp"
        ErrorLog /var/graphite/log/webapp/error.log
        CustomLog /var/graphite/log/webapp/access.log common

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/local/etc/graphite/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        # XXX You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /opt/local/etc/graphite/graphite.wsgi

        Alias /content/ /opt/local/graphite/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "/opt/local/lib/python2.7/site-packages/django/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /opt/local/etc/graphite/>
                Order deny,allow
                Allow from all
        </Directory>

        <Directory /opt/local/graphite/webapp/content/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>

Start Services

svcadm enable apache
carbon-cache.py --config /opt/local/etc/graphite/carbon.conf start

Test

Go to YOURIP/FQDN:80. You should see the Graphite default dashboard. Or a Django error if you're unlucky.

Install and Configure Panoptimon with Graphite

pkgin in ruby193 scmgit ruby193-eventmachine ruby193-daemons ruby193-thin gcc47 gmake

cd /opt
git clone https://github.com/synthesist/panoptimon.git

mkdir -p /opt/fire/etc/panoptimon/{collectors,plugins}
cd /opt/fire/etc/panoptimon

Edit /opt/fire/etc/panoptimon/panoptimon.json

{
  "daemonize": "false",
  "collector_interval": 10,
  "collector_timeout": 120
}

Set up your collectors

cd /opt/fire/etc/panoptimon/collectors
for x in cpu disk iostat load memory service zfs ; do ln -s /opt/panoptimon/collectors/$x . ; echo "{}" > $x.json ; done

Set up your plugins

cd /opt/fire/etc/panoptimon/plugins
for x in graphite metrics_http status_http ; do ln -s /opt/panoptimon/plugins/$x . ; echo "{}" > $x.json ; done

Edit /opt/fire/etc/panoptimon/collectors/service.json

{
  "faults": true,
  "interval": 60,
  "flaptime": 30,
  "since": 900,
  "services": {
    "smf": { "-monitor": [ "*" ] }
  }
}

Setting faults to true sets a field which returns the number of faulted services. Handy for simple alerting.

Edit /opt/fire/etc/panoptimon/plugins/graphite.json

{
  "host": "localhost",
  "prefix": "host.<%= host %>"
}

Start Pan

env RUBYLIB=/opt/panoptimon/lib /opt/panoptimon/bin/panoptimon -C /opt/fire/etc/panoptimon -D

Check Metrics

Have a look at http://YOURIP:8080. You should see Pan's status page. http://YOURIP:8080/metrics.json services metrics in ... JSON.

Go back to your Graphite console. Expand the Graphite folder. Under that there should be a hosts subfolder. Expand that, and your test systems's hostname should list the Pan collectors being emitted to Graphite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment