Skip to content

Instantly share code, notes, and snippets.

@LotharSee
Last active August 29, 2015 14:14
Show Gist options
  • Save LotharSee/77a69f4e5c37643b50ee to your computer and use it in GitHub Desktop.
Save LotharSee/77a69f4e5c37643b50ee to your computer and use it in GitHub Desktop.
Non-root Datadog Agent supervisor on CentOS
#!/bin/sh
###############################################################################
# datadog-agent
#
# Inspired by Boxed Ice <[email protected]>
# Forked by Datadog, Inc. <[email protected]>
#
# Licensed under Simplified BSD License (see LICENSE)
#
###############################################################################
#
# chkconfig: 345 85 15
# description: Datadog Monitoring Agent
### BEGIN INIT INFO
# Provides: datadog-agent
# Short-Description: Start and start datadog-agent
# Description: datadog-agent is the monitoring agent component for Datadog
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
AGENTPATH="/opt/datadog-agent/agent/agent.py"
AGENTCONF="/etc/dd-agent/datadog.conf"
DOGSTATSDPATH="/opt/datadog-agent/agent/dogstatsd.py"
AGENTUSER="dd-agent"
FORWARDERPATH="/usr/bin/dd-forwarder"
SUPERVISORD_PATH="PATH=/opt/datadog-agent/embedded/bin:/opt/datadog-agent/bin:$PATH /opt/datadog-agent/bin/supervisord"
SUPERVISORCTL_PATH="/opt/datadog-agent/bin/supervisorctl"
SUPERVISOR_CONF="/etc/dd-agent/supervisor.conf"
SUPERVISOR_SOCK="/tmp/datadog-supervisor.sock"
# Source function library.
. /etc/rc.d/init.d/functions
PROG="datadog-agent"
SUPERVISOR_PIDFILE=/tmp/datadog-supervisord.pid
LOCKFILE=/tmp/$PROG
check_status() {
# run checks to determine if the service is running or use generic status
# if the sock exists, we can use supervisorctl
if [ -e $SUPERVISOR_SOCK ]; then
s=`$SUPERVISORCTL_PATH -c $SUPERVISOR_CONF status`
# number of RUNNING supervisord programs (ignoring pup and dogstatsd)
p=`echo "$s" | grep -v dogstatsd | grep -c RUNNING`
# number of expected running supervisord programs (ignoring pup and dogstatsd)
c=`grep -v dogstatsd $SUPERVISOR_CONF | grep -c '\[program:'`
if [ "$p" -ne "$c" ]; then
echo "$s"
echo -n "Datadog Agent (supervisor) is NOT running all child processes"; failure; echo
return 1
else
echo -n "Datadog Agent (supervisor) is running all child processes"; success; echo
return 0
fi
else
# if no sock, use the rc status function
status -p $SUPERVISOR_PIDFILE $PROG
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
echo -n "Datadog Agent (supervisor) is running."; success; echo
else
echo -n "Datadog Agent (supervisor) is NOT running."; failure; echo
fi
return $RETVAL
fi
if [ -f "$LOCKFILE" ]; then
echo -n 'Datadog Agent is running'; success; echo
return 0
else
echo -n 'Datadog Agent is NOT running'; failure; echo
return 1
fi
}
grab_status() {
GRABSTATUS=`check_status &>/dev/null`
}
start() {
if [ ! -f $AGENTCONF ]; then
echo "$AGENTCONF not found. Exiting."
exit 3
fi
su $AGENTUSER -c "$AGENTPATH configcheck" > /dev/null
if [ $? -ne 0 ]; then
echo -n $'\n'"Invalid check configuration. Please run sudo /etc/init.d/datadog-agent configtest for more details."
echo -n $'\n'"Resuming starting process."$'\n'
fi
# no need to test for status before daemon,
# the daemon function does the right thing
echo -n "Starting Datadog Agent (using supervisord):"
daemon --user=$AGENTUSER --pidfile=$SUPERVISOR_PIDFILE $SUPERVISORD_PATH -c $SUPERVISOR_CONF > /dev/null
# check if the agent is running once per second for 10 seconds
retries=10
while [ $retries -gt 1 ]; do
if grab_status; then
touch $LOCKFILE
success; echo
return 0
else
retries=$(($retries - 1))
sleep 1
fi
done
# after 10 tries the agent didn't start. Report the error and stop
echo; check_status # check status will show us the error and take care of calling `failure`
stop
return 1
}
stop() {
# no need to test for status before killproc,
# it does the right thing. and testing supervisorctl status
# before killproc can lead to states where you cannot stop!
echo -n 'Stopping Datadog Agent (using killproc on supervisord): '
killproc -p $SUPERVISOR_PIDFILE
rm -f $LOCKFILE
echo
return 0
}
restart() {
stop
start
}
info() {
shift # Shift 'info' out of the args so we can pass any
# additional options to the real command
# (right now only dd-agent supports additional flags)
su $AGENTUSER -c "$AGENTPATH info $@"
COLLECTOR_RETURN=$?
su $AGENTUSER -c "$DOGSTATSDPATH info"
DOGSTATSD_RETURN=$?
su $AGENTUSER -c "$FORWARDERPATH info"
FORWARDER_RETURN=$?
exit $(($FORWARDER_RETURN+$COLLECTOR_RETURN+DOGSTATSD_RETURN))
}
configcheck() {
su $AGENTUSER -c "$AGENTPATH configcheck"
exit $?
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
check_status
;;
info)
info "$@"
;;
configcheck)
configcheck
;;
configtest)
configcheck
;;
jmx)
shift
su $AGENTUSER -c "$AGENTPATH jmx $@"
exit $?
;;
*)
echo "Usage: $0 {start|stop|restart|info|status|configcheck|configtest|jmx}"
exit 2
esac
exit $?
29d28
< PIDPATH="/var/run/dd-agent/"
34c33
< SUPERVISOR_SOCK="/var/tmp/datadog-supervisor.sock"
---
> SUPERVISOR_SOCK="/tmp/datadog-supervisor.sock"
40,42c39,40
< PIDFILE=$PIDPATH/$PROG.pid
< SUPERVISOR_PIDFILE=/var/run/datadog-supervisord.pid
< LOCKFILE=/var/lock/subsys/$PROG
---
> SUPERVISOR_PIDFILE=/tmp/datadog-supervisord.pid
> LOCKFILE=/tmp/$PROG
106c104
< daemon --pidfile=$SUPERVISOR_PIDFILE $SUPERVISORD_PATH -c $SUPERVISOR_CONF > /dev/null
---
> daemon --user=$AGENTUSER --pidfile=$SUPERVISOR_PIDFILE $SUPERVISORD_PATH -c $SUPERVISOR_CONF > /dev/null

Procedure to make the Datadog Agent Supervisor run as dd-agent instead of root. The main issue is file permission. I moved lock, pid and sock to /tmp to ensure it works well. Any other directory with rw permission for the user dd-agent would make it.

  • Stop the Agent. service datadog-agent stop
  • Be sure that all processes are stopped and the pid/sock got removed
    • Check ps aux | grep -v grep | grep datadog should be empty
    • Remove /var/run/datadog-supervisord.pid, /var/tmp/datadog-supervisor.sock, /var/lock/subsys/datadog-agent.
  • Give permission to dd-agent to the supervisor log file.
    • Run chown dd-agent:dd-agent /var/log/datadog/supervisord.log
  • Update the init script and the supervisor configuration to put files in tmp.
    • Replace /etc/init.d/datadog-agent and /etc/dd-agent/supervisor.conf with the ones provided in this Gist.
  • Start the Agent. service datadog-agent start. Now you should see the supervisor running with the user dd-agent!
ps aux | grep supervisord
dd-agent  4340  0.0  2.0 194172 10288 ?        Ss   11:46   0:00 /opt/datadog-agent/embedded/bin/python /opt/datadog-agent/bin/supervisord -c /etc/dd-agent/supervisor.con

Done on CentOS 7, but it should work with any rpm-based system. And the idea would be the same for any apt-based system.

[supervisorctl]
serverurl = unix:///tmp/datadog-supervisor.sock
[unix_http_server]
file=/tmp/datadog-supervisor.sock
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisord]
http_port = /tmp/datadog-supervisor.sock
minfds = 1024
minprocs = 200
loglevel = info
logfile = /var/log/datadog/supervisord.log
logfile_maxbytes = 50MB
nodaemon = false
pidfile = /tmp/datadog-supervisord.pid
logfile_backups = 10
environment=PYTHONPATH=/opt/datadog-agent/agent:/opt/datadog-agent/agent/checks,LANG=POSIX
[program:collector]
command=/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/agent.py foreground --use-local-forwarder
stdout_logfile=NONE
stderr_logfile=NONE
priority=999
startsecs=5
startretries=3
user=dd-agent
environment=PYTHONPATH='/opt/datadog-agent/agent:/opt/datadog-agent/agent/checks/libs:$PYTHONPATH'
[program:forwarder]
command=/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/ddagent.py
stdout_logfile=NONE
stderr_logfile=NONE
startsecs=5
startretries=3
priority=998
user=dd-agent
[program:dogstatsd]
command=/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/dogstatsd.py --use-local-forwarder
stdout_logfile=NONE
stderr_logfile=NONE
startsecs=5
startretries=3
priority=998
user=dd-agent
[group:datadog-agent]
programs=forwarder,collector,dogstatsd
2c2
< serverurl = unix:///var/tmp/datadog-supervisor.sock
---
> serverurl = unix:///tmp/datadog-supervisor.sock
5c5
< file=/var/tmp/datadog-supervisor.sock
---
> file=/tmp/datadog-supervisor.sock
11c11
< http_port = /var/tmp/datadog-supervisor.sock
---
> http_port = /tmp/datadog-supervisor.sock
18c18
< pidfile = /var/run/datadog-supervisord.pid
---
> pidfile = /tmp/datadog-supervisord.pid
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment