Installing graphite 0.10, collectd and grafana on centos 6

graphite.md

Installing Graphite:

Graphite does two things:

1. Store numeric time-series data
2. Render graphs of this data on demand

What Graphite does not do is collect data for you, however there are some tools out there that know how to send data to graphite. Even though it often requires a little code, sending data to Graphite is very simple.

Architecture:

Graphite consists of 3 software components:

• carbon - a Twisted daemon that listens for time-series data
• whisper - a simple database library for storing time-series data (similar in design to RRD)
• graphite webapp - A Django webapp that renders graphs on-demand using Cairo

Installing:

Install Required repo:

Redhat:

curl -o epel.rpm -L http://download.fedoraproject.org/pub/epel/6/$(arch)/epel-release-6-8.noarch.rpm
rpm -ivh epel.rpm


yum install pycairo Django14 python-ldap python-memcached python-sqlite2 bitmap bitmap-fonts-compat \
python-devel python-crypto pyOpenSSL gcc python-zope-filesystem python-zope-interface git gcc-c++ \
zlib-static MySQL-python python-txamqp python-setuptools python-psycopg2 mod_wsgi

Ubuntu:

apt-get install python-cairo python-twisted python-django python-django-tagging python-ldap \
python-memcache python-sqlite python-simplejson python-txamqp ca-certificates

Download and extract packages:

carbon_version="0.9.12"
whisper_version="0.9.12"
graphite_web_version="0.9.12"
build_dir="/opt"
cd ${build_dir}
curl -s -L https://github.com/graphite-project/carbon/archive/${carbon_version}.tar.gz | tar xz
curl -s -L https://github.com/graphite-project/whisper/archive/${whisper_version}.tar.gz | tar xz
curl -s -L https://github.com/graphite-project/graphite-web/archive/${graphite_web_version}.tar.gz | tar xz
easy_install django-tagging==0.3.1
easy_install twisted==11.1.0
easy_install txamqp==0.4

Install webapp (this requires djando-tagging)

cd /opt/graphite-web-0.9.12
python setup.py install # subscribe to untar webapp

Install carbon (this requires twisted)

cd /opt/carbon-0.9.12
python setup.py install

Install whisper (this requires twisted)

cd /opt/whisper-0.9.12
python setup.py install

Configuring:

Install postgres as a backend for graphite:

yum -y install postgresql postgresql-server postgresql-devel

Configure postgres to allow graphite user:

service postgresql initdb
grep "listen_addresses = '0.0.0.0'" /var/lib/pgsql/data/postgresql.conf
if [[ $? -ne 0 ]]; then
  echo "listen_addresses = '0.0.0.0'" >> /var/lib/pgsql/data/postgresql.conf
fi
service postgresql start
sudo -u postgres psql template1 <<END
create user graphite with password 'graphite';
create database graphite with owner graphite;
END

Configure graphite settings:

cat > /opt/graphite/webapp/graphite/local_settings.py <<EOF
SECRET_KEY = 'UNSAFE_DEFAULT'

TIME_ZONE = 'GMT'

#DOCUMENTATION_URL = "http://graphite.readthedocs.org/"

# Logging
#LOG_RENDERING_PERFORMANCE = True
#LOG_CACHE_PERFORMANCE = True
#LOG_METRIC_ACCESS = True

# Enable full debug page display on exceptions (Internal Server Error pages)
#DEBUG = True

# If using RRD files and rrdcached, set to the address or socket of the daemon
#FLUSHRRDCACHED = 'unix:/var/run/rrdcached.sock'

# Change only GRAPHITE_ROOT if your install is merely shifted from /opt/graphite
# to somewhere else
#GRAPHITE_ROOT = '/opt/graphite'

# Most installs done outside of a separate tree such as /opt/graphite will only
# need to change these three settings. Note that the default settings for each
# of these is relative to GRAPHITE_ROOT
#CONF_DIR = '/opt/graphite/conf'
#STORAGE_DIR = '/opt/graphite/storage'
#CONTENT_DIR = '/opt/graphite/webapp/content'

# To further or fully customize the paths, modify the following. Note that the
# default settings for each of these are relative to CONF_DIR and STORAGE_DIR
#
## Webapp config files
#DASHBOARD_CONF = '/opt/graphite/conf/dashboard.conf'
#GRAPHTEMPLATES_CONF = '/opt/graphite/conf/graphTemplates.conf'

## Data directories
# NOTE: If any directory is unreadable in DATA_DIRS it will break metric browsing
#WHISPER_DIR = '/opt/graphite/storage/whisper'
#RRD_DIR = '/opt/graphite/storage/rrd'
#DATA_DIRS = [WHISPER_DIR, RRD_DIR] # Default: set from the above variables
#LOG_DIR = '/opt/graphite/storage/log/webapp'
#INDEX_FILE = '/opt/graphite/storage/index'  # Search index file


DATABASES = {
   'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'graphite',
        'USER': 'graphite',
        'PASSWORD': 'graphite',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}
EOF

Modify PostgreSQL "Client Authentication" -- use md5:

cat > /var/lib/pgsql/data/pg_hba.conf <<EOM
# "local" is for Unix domain socket connections only
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5
EOM

Restart PostrgreSQL to take effect:

service postgresql restart

Initialize Django DB creation (requires configuring /opt/graphite/webapp/graphite/local_settings.py):

cd /opt/graphite/webapp/graphite
python manage.py syncdb --noinput

Change the permission for webapp storage to httpd user

chown -R apache:apache /opt/graphite/storage/
chcon -R -h -t httpd_sys_content_t /opt/graphite/storage

Configure graphite web:

cat > /opt/graphite/conf/graphite.wsgi <<EOF
import os, sys
sys.path.append('/opt/graphite/webapp')
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

# READ THIS
# Initializing the search index can be very expensive, please include
# the WSGIImportScript directive pointing to this script in your vhost
# config to ensure the index is preloaded before any requests are handed
# to the process.
from graphite.logger import log
log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
import graphite.metrics.search
EOF

# Configuring whisper

cat > /opt/graphite/conf/storage-schemas.conf <<EOF
# Schema definitions for whisper files. Entries are scanned in order,
# and first match wins.
[default]
pattern = .*
retentions = 1s:30m,1m:1d,5m:2y
EOF

cat > /opt/graphite/conf/carbon.conf <<EOF
[cache]
ENABLE_LOGROTATION = True
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
LOG_LISTENER_CONNECTIONS = True
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
CARBON_METRIC_INTERVAL = 60
ENABLE_AMQP = False

[relay]
ENABLE_LOGROTATION = True
USER =
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
LOG_LISTENER_CONNECTIONS = True
RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
CARBON_METRIC_INTERVAL = 60

[aggregator]
ENABLE_LOGROTATION = True
USER =
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
LOG_LISTENER_CONNECTIONS = True
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5
CARBON_METRIC_INTERVAL = 60
EOF

cat > /opt/graphite/conf/storage-aggregation.conf <<EOF
[min]
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.max$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.5
aggregationMethod = average
EOF

On some systems, hostname -f can not work due to incorrect /etc/hosts entry. Fix:

if ! $(hostname -f > 2&>1 /dev/null); then
  cat > /etc/hosts <<EOF
127.0.0.1   $(hostname) localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
EOF
fi

Configuring httpd:

cat > /etc/httpd/conf.d/graphite.conf <<EOF
<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

WSGISocketPrefix run/wsgi

<VirtualHost *:80>
        ServerName $(hostname -f)
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /var/log/httpd/graphite_error.log
        CustomLog /var/log/httpd/graphite_access.log common

        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

        Alias /content/ /opt/graphite/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        <Directory /opt/graphite/conf/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>
EOF

cat > /etc/init.d/carbon-cache <<\EOF
#!/bin/bash
#
# This is used to start/stop the carbon-cache daemon

# chkconfig: - 99 01
# description: Starts the carbon-cache daemon

# Source function library.
. /etc/init.d/functions

RETVAL=0
prog="carbon-cache"

start_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

start_cache () {
     /usr/bin/python /opt/graphite/bin/carbon-cache.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_cache () {
          /usr/bin/python /opt/graphite/bin/carbon-cache.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

# See how we were called.
case "$1" in
  start)
    #start_relay
    start_cache
        ;;
  stop)
    #stop_relay
    stop_cache
        ;;
  restart)
    #stop_relay
    stop_cache
    sleep 5
    #start_relay
    start_cache
    ;;

  *)
        echo $"Usage: $0 {start|stop}"
        exit 2
        ;;
esac

EOF
chmod +x /etc/init.d/carbon-cache

service carbon-cache start
service httpd start

Install collectd:

yum install -y rrdtool rrdtool-devel perl rrdtool-perl libgcrypt-devel gcc make gcc-c++ perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-ExtUtils-Embed
cd /opt
curl -s -L http://collectd.org/files/collectd-5.4.0.tar.bz2 | tar jx
cd collectd-5.4.0
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib --mandir=/usr/share/man --enable-cpu --enable-curl --enable-df --enable-exec --enable-load --enable-logfile --enable-memory --enable-network --enable-nginx --enable-syslog  --enable-rrdtool --enable-uptime --enable-write_graphite
make
make install
cp contrib/redhat/init.d-collectd /etc/init.d/collectd
chmod 755 /etc/init.d/collectd
chown root:root /etc/init.d/collectd
/etc/init.d/collectd start
chkconfig collectd on

Configure collectd to write system metrics to graphite by editing /etc/collectd.conf and adding the following lines:

LoadPlugin syslog

LoadPlugin "logfile"
<Plugin "logfile">
  LogLevel "info"
  File "/var/log/collectd.log"
  Timestamp true
</Plugin>

LoadPlugin cpu
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin rrdtool

LoadPlugin write_graphite
<Plugin "write_graphite">
 <Carbon>
   Host "127.0.0.1"
   Port "2003"
   Prefix "collectd."
   #Postfix ""
   Protocol "tcp"
   EscapeCharacter "_"
   SeparateInstances true
   StoreRates false
   AlwaysAppendDS false
 </Carbon>
</Plugin>

Restart collectd to start sending metrics to graphite

/etc/init.d/collectd restart

Installing Grafana

Grafana is a custom dashboard for graphite.

cd /opt
mkdir grafana
cd grafana
curl -s -L https://github.com/torkelo/grafana/archive/v1.4.0.tar.gz | tar xz

Configure grafana with httpd:

cat > /etc/httpd/conf.d/grafana.conf <<EOF
Alias /grafana /opt/grafana

<Location /grafana>
 Order deny,allow
 Allow from 127.0.0.1
 Allow from ::1
 Allow from all
</Location>
EOF

Restart httpd to refresh graphite:

service httpd restart

Thanks so much.
Detailed procedure to get everything working in the first shot

I had a problem in getting the graph in graphite. On analysis, I found out that whisper database was not getting populated with the value. Digging deep, I understood there could be possible mismatch with the data interval configured in the collectd and whisper.

Did the following

1. Configured the /etc/collectd.conf to say interval has 60 seconds [Add the line Interval 60 in the file]
2. Configure the graphite schema [/opt/graphite/conf/storage-schemas.conf]
   [carbon]
   pattern = ^carbon.
   retentions = 60:90d

Rebuild the RRD and whisper database by removing all the *.rrd and *.wsp files and doing the service restart

1. /etc/init.d/collectd restart
2. service carbon-cache restart
   3 service httpd restart

Thanks for writing this up!

Using CentOS 6.6, I had to make two changes:

1. Remove the line that updates /etc/postgresql/9.*/main/pg_hba.conf (that file is actually located within the /var/lib/pgsql/data/ directory, and the rest of the procedure worked when omitting that step entirely).
2. Restart PostgreSQL after the 'Modify PostgreSQL "Client Authentication" -- use md5' step.

Here's the fork:

https://gist.github.com/mrideout/af84092f6b97ab53fb37

Here are the two updates:

https://gist.github.com/mrideout/af84092f6b97ab53fb37/revisions

Feel free free to pull it back.

Matt

Any idea why the graphite browser would pull up in a browser, but everything below the nav bar is blank?

Specifically, I can log into graphite, but I can't edit/modify/create graphs. Existing graphs continue to collect data and update, since I pasted the image links into a separate noc page.

This is on CentOS 6.6

I'm seeing a similar issue to what raysanders is describing.

regarding the "Installation of Grafana", is that all? Any additional actions?

In Redhat 6 You should add :

service httpd stop
service postgresql stop
setsebool -P httpd_can_network_connect 1
service httpd start
service postgresql start

For this error : " could not connect to server: Permission denied " If Graphite-Web didn't work

@s2deaths
Thanks for the fix

CentOS 6.5 - I was getting an error in the /etc/httpd/logs/graphite_error.log

[Tue Dec 15 16:08:54 2015] [error] [client 172.26.253.94] File "/usr/lib/python2.6/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
[Tue Dec 15 16:08:54 2015] [error] [client 172.26.253.94] self.connection = Database.connect(**conn_params)
[Tue Dec 15 16:08:54 2015] [error] [client 172.26.253.94] OperationalError: could not connect to server: Permission denied
[Tue Dec 15 16:08:54 2015] [error] [client 172.26.253.94] \tIs the server running on host "localhost" and accepting
[Tue Dec 15 16:08:54 2015] [error] [client 172.26.253.94] \tTCP/IP connections on port 5432?

Fixed with "setsebool -P httpd_can_network_connect_db on"