Additional Information about Monitoring Riak

Riak is a complex system that includes many moving parts to monitor, such as the health of the hardware, the well-being of the software, and the responsiveness of the network. This document will discuss metrics, thresholds, and values that indicate when your monitoring system should be sending alarms.

System metrics to monitor

Metric	Threshold
CPU	75% * number of cores
Memory	70% - buffers
Disk Space	75%
Network	70% sustained
File Descriptors	75% of ulimit
Swap	> 0 KB

Log Events to Alert On

String	Log File	Reason
`eaccess`	console.log	File/Directory Permissions Issue
`emfile`	console.log	Exhausted File Handles
`erofs`	console.log	File System Mounted in Read-Only Mode
`noproc`	console.log	Unexpectedly Missing Process
`undef`	console.log	Missing/Incorrect Erlang Modules
`system_limit`	console.log	Erlang VM Resource Exhaustion
`Compaction error`	LevelDB LOGs	LevelDB Compaction Error
`waiting`	LevelDB LOGs	LevelDB Stalls

found in http://docs.basho.com/riak/latest/ops/running/recovery/errors/

Riak Nagios Project

Located at http://github.com/basho_labs/riak_nagios. Provided tests include:

check_connection_pools
check_file_handle_count
check_leveldb_compaction
check_node
check_node_up
check_port_count
check_riak_kv_up
check_riak_repl

Relevant Blog Posts on Monitoring

Acreman, S. (2014, April 12). Montoring Riak. [Blog post] Retrieved from http://blog.dataloop.io/2014/04/12/monitoring-riak/

Discussion of using CollectD to consume all of the statistics provided by Riak
Santero, T. (2013, March 27). Monitoring Riak. [Slide deck] Retrieved from https://speakerdeck.com/tsantero/monitoring-riak

Provides information about monitoring suggestions, including system limits and monitoring tools.

Riak-Admin Stat (Exometer) [Riak 2.0+ only]

Listing all of the statistics and their values

$ riak-admin stat show '*.**'

Note: The single ticks are necessary to prevent the shell from attempting to interpret the '*' as a globbing operator

Querying for a specific statistic

$ riak-admin stat show riak.riak_kv.vnode.gets  
 [riak,riak_kv,vnode,gets]: [{count,0},{one,0}]

Querying for child statistics

$ riak-admin stat show 'riak.riak_kv.vnode.gets.*'  
 [riak,riak_kv,vnode,gets,time]: [{n,0},{mean,0},{min,0},{max,0},{median,0},{50,0},{75,0},{90,0},{95,0},{99,0},{999,0}]

Note: The single ticks are necessary to prevent the shell from attempting to interpret the '*' as a globbing operator

Querying for a subtree

$ riak-admin stat show 'riak.riak_kv.vnode.gets.**'  
 [riak,riak_kv,vnode,gets]: [{count,0},{one,0}]
 [riak,riak_kv,vnode,gets,time]: [{n,0},{mean,0},{min,0},{max,0},{median,0},{50,0},{75,0},{90,0},{95,0},{99,0},{999,0}]

Note: The single ticks are necessary to prevent the shell from attempting to interpret the '*' as a globbing operator

Getting a list of all statistics and their type

$ riak-admin stat info -type '*.**'
 [riak,common,cpu_stats]: type = cpu
 [riak,common,mem_stats]: type = function
 [riak,common,memory_stats]: type = function
 [riak,riak_api,pbc_connects]: type = spiral
 [riak,riak_api,pbc_connects,active]: type = function
 [riak,riak_core,converge_delay]: type = duration
 [riak,riak_core,dropped_vnode_requests_total]: type = counter
 [riak,riak_core,gossip_received]: type = spiral
 [riak,riak_core,handoff_timeouts]: type = counter
 [riak,riak_core,ignored_gossip_total]: type = counter
 [riak,riak_core,rebalance_delay]: type = duration
 ...

This output could be used as input to a script that generates a CollectD configuration.

binarytemple/MonitoringRiak.md