Basic jenkins monitoring

.gitignore

/zbx_env

README.md

Quick zabbix setup without all the fuss.

Deploys a basic setup locally which is good enough to monitor the Jenkins infrastructure. This is a heavily modified form of https://github.com/zabbix/zabbix-docker

Rationale

Some of the metrics that are we care about like the time taken to spawn a slave node is easiest obtained with a Jenkins file. Also,

1. Tests the pipeline library, k8s java libs
2. No need to write Java
3. No need to understand the plugin architecture for now
4. Time taken for some ops might be too long for a classic HTTP GET in a loop
5. zabbix_send is pretty easy to understand once it works

The top level idea is to compute relevant values in a Jenkinsfile, send it with zabbix_send and run the job under cron.

Bring up the services

$ docker-compose up

Login to the web UI

Login to http://0.0.0.0:8081 with creds Admin/zabbix

Configure

Add a host at 'Configuration > Hosts > Create Host'
    Host name        : rivendell
    Groups           : Linux Server worked for me.
    Agent Interfaces : 127.0.0.1 / IP / 10051

Add an application
    Name             : Jenkins

Add an Item
    Name             : k8s.connect.time
    Key              : k8s.connect.time
    Type             : Zabbix Trapper
    Allowed hosts    : Your IP.
    Application      : Jenkins

Send Data

Should be able to send data right now using the CLI

$ zabbix_sender -z '127.0.0.1' -p 10051 -s rivendell -k k8s.connect.time -o 240

Debug

High chance that something will go silly and this wont work. Logs can be viewed with $ docker logs --follow zabbixdocker_zabbix-server_1 and it usually contains something useful.

Help!

Please update this if you find any issues.

docker-compose.yaml

version: '2'
services:
  mysql-server:
    image: mysql:5.7
    command: [mysqld, --character-set-server=utf8, --collation-server=utf8_bin]
    volumes:
      - ./zbx_env/var/lib/mysql:/var/lib/mysql:rw
      - ./zbx_env/usr/lib/mysql:/usr/lib/mysql:rw
    env_file:
      - env
    user: root
    networks:
      zabbix:

  zabbix-server:
    image: zabbix/zabbix-server-mysql:ubuntu-3.4-latest
    ports:
      - "10051:10051"
    volumes:
      - ./zbx_env/usr/lib/zabbix:/usr/lib/zabbix:rw
    ulimits:
      nproc: 65535
      nofile:
        soft: 20000
        hard: 40000
    mem_limit: 512m
    env_file:
      - env
    user: root
    networks:
      zabbix:

  zabbix-web-nginx-mysql:
    image: zabbix/zabbix-web-nginx-mysql:ubuntu-3.4-latest
    ports:
      - "8081:80"
      - "8443:443"
    links:
      - mysql-server:mysql-server
      - zabbix-server:zabbix-server
    mem_limit: 512m
    volumes:
      - ./zbx_env/etc/ssl/nginx:/etc/ssl/nginx:ro
    env_file:
      - env
    user: root
    networks:
      zabbix:

networks:
  zabbix:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    ipam:
      driver: default
      config:
      - subnet: 172.16.238.0/24
        gateway: 172.16.238.1

env

DB_SERVER_HOST=mysql-server
DB_SERVER_PORT=3306
MYSQL_DATABASE=zabbix
MYSQL_PASSWORD=zabbix
MYSQL_ROOT_PASSWORD=root_pwd
MYSQL_USER=zabbix
ZBX_HOSTNAME=zabbix-server
ZBX_SERVER_HOST=zabbix-server

Jenkinsfile

#!/usr/bin/env groovy
@Library('github.com/fabric8io/fabric8-pipeline-library@master') _

pipeline {
    agent any

    triggers {
        // Run this every 5 minutes or so.
        cron('H/5 * * * *')
    }

    stages {
        // Assumption: Agent any should start this task on the master and we
        // haven't had many issues on this.

        stage('Booting up') {
            steps {
                sh 'hostname'
                // sh 'docker -v'

                // Doing this with a script block in Groovy surprisingly turned
                // out to be much harder than it should be.
                //
                // Time in nanoseconds, that's as much precision as we can get.
                sh 'date +%s%N > start.time'
            }
        }

        stage('Slave') {
            // Do something on the slave node. This often fails and we don't
            // completely know why.

            steps {
                container('ui') {
                    steps {
                        sh 'hostname'
                    }
                }
            }
        }

        stage('Report to Zabbix') {
            steps {
                sh 'date +%s%N > end.time'
                sh 'echo print $(cat end.time) - $(cat start.time) | python > total.time'

                // Report the time taken to _mostly_ spawn a slave node to Zabbix.
                sh 'zabbix_sender -vv -z zabbix.devshift.net -p 10051 -s test.cd.fabric8.io -k test.jenkins.fabric8.xyz -o - < total.time'
            }
        }
    }
}

Makefile

.PHONY: hook
hook:
	ln -sf ../../post-commit.sh .git/hooks/post-commit.sh

post-commit.sh

#!/bin/sh

# The SCM Poll + github webhook approach repeatedly failed because Jenkins kept
# disabling it again and again.
# curl http://localhost:8090/git/notifyCommit\?url\=file:///home/j/tmp/Jenkins

echo "Building Jenkins job with commit hook"

curl http://jenkins.myproject.`minishift ip`.nip.io/job/Heimdall/build