Wildcard munin plugin to monitor filesystem usage

munin-plugin.disks_

#!/bin/bash

: << =cut
=head1 NAME

disks_ - Wildcard plugin to monitor filesystem usage

=head1 CONFIGURATION

This plugin does not normally require configuration. However, you can tweak its
behavior by using the following environment variables

 df             - df program to use
 df_args        - Arguments to df
 exclude        - Space separated list of filesystem types to exclude
 exclude_re     - Regex to match device or mountpoint paths to exclude
 include_re     - Regex to match device or mountpoint paths to include
 warning        - Warning percentage
 critical       - Critical percentage
 percent        - Show percentages instead of absolute values [auto|on|off]
 total          - Enable/disable graph total [auto|on|off]
 persistence    - Time string to remove dangling filesystems
 unknown_limit  - Number of tolerable unknown values
 mounts         - Path to mounts file

See the 'EXAMPLE CONFIGURATION' section below for more info about how to use
the environment variables.

This plugin requires PhrozenByte's statefile helper. Just download the script
from <https://daniel-rudolf.de/oss/munin-statefile-helper> and copy it to
'/usr/share/munin/plugins/plugin-statefile.sh' (see '\$MUNIN_LIBDIR').

=head2 WILDCARD CONFIGURATION

This plugin is a wildcard plugin using the form \`disks_<mode>\`. You can
replace '<mode>' by one of the following

 abs        - Monitors the absolute filesystem usage in bytes
 pct        - Monitors the relative filesystem usage in percent
 inode_abs  - Monitors the absolute inode usage in number of inodes
 inode_pct  - Monitors the relative inode usage in percent

This plugin supports both the 'autoconf' and 'suggest' capability.

=head2 DEFAULT CONFIGURATION

 [disks_*]
  env.df            df
  env.df_args       
  env.exclude       none unknown rootfs iso9660 squashfs udf romfs ramfs tmpfs debugfs cgroup_root devtmpfs
  env.exclude_re    
  env.include_re    
  env.warning       92
  env.critical      98
  env.percent       auto
  env.total         auto
  env.persistence   -6 hours
  env.unknown_limit 3
  env.mounts        /proc/mounts

=head2 EXAMPLE CONFIGURATION

If you want to monitor local filesystems only, use the following configuration

 [disks_*]
  env.df_args -l

This plugin automatically determines whether it's supposed to show the
filesystem or inode usage. If you want to explicitly show inode usage, add the
'-i' option to the 'df_args' environment variable; see below

 [disks_*]
  env.df_args -i

If you want to exclude all filesystems mounted below '/var/backups', try the
following

 [disks_*]
  env.exclude_re ^/var/backups/

Munin will send notifications as soon as one of your filesystems exceeds 92%
usage (resp. 98% usage for critical warnings). You can either configure these
limits on a global or a per-filesystem basis. To set the warning limit of the
'/dev/sda1' device mounted at '/boot' to 80%, try the following configuration

 [disks_*]
  env._dev_sda1__boot_warning 80

This plugin automatically determines whether it's supposed to show percentages
or absolute values (bytes/inodes) based on the plugin's basename. If you want
to always show percentages, try the following configuration. If you rather want
to never show percentages, replace 'on' by 'off'.

 [disks_*]
  env.percent on

When monitoring the absolute usage of a filesystem in bytes, a total line is
added to your graph automatically. If you want to manually disable the total
line, use the following configuration. If you rather want to force enable the
total line, try 'on' instead of 'off'.

 [disks_*]
  env.total off

This plugin supports persistency. If a filesystem disappears, this plugin still
considers the filesystem, but reports a unknown ('U') value. Munin won't send
notifications until 'unknown_limit' consecutive runs of the plugin, defaulting
to 3 runs (plugins are usually run every 5 minutes). Dangling filesystems will
be removed after a given period of time ('persistence'), defaulting to 6 hours.
You can change both the number of acceptable unknown values ('unknown_limit'
environment variable) and after which time dangling filesystems should be
removed ('persistence' environment variable). A dangling filesystem is
considered in the 'total' line with the last known usage until it is removed.

For example, if you don't mind a filesystem disappearing for no longer than
an hour, set 'unknown_limit' to 12 runs. Since you usually don't change
disks often, dangling filesystems should be removed after 1 day. Try the
following configuration

 [disks_*]
  env.persistence -1 day
  env.unknown_limit 12

To disable persistency (i.e. silently remove filesystems), try the following

 [disks_*]
  env.persistence now
  env.unknown_limit 0

=head1 REPLACES

This plugin is a drop-in replacement for munin's default \`df\`, \`df_abs\`
and \`df_inode\` plugins. This plugin uses a more error-prone approach and
supports additional features. However, unfortunately due to bugs in munin's
default plugins you can't inherit monitoring data.

=head1 AUTHOR

Copyright (C) 2019 Daniel Rudolf

=head1 LICENSE

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3 of the License only.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

See <http://www.gnu.org/licenses/> to receive a full-text-copy of
the GNU General Public License.

=head1 MAGIC MARKERS

 #%# family=auto
 #%# capabilities=autoconf suggest

=cut

if ! [ -f "$MUNIN_LIBDIR/plugins/plugin-statefile.sh" ]; then
    [ "$1" != "autoconf" ] || { echo "no ('$MUNIN_LIBDIR/plugins/plugin-statefile.sh' not found)"; exit 0; }

    echo "Unable to include statefile helper from '$MUNIN_LIBDIR/plugins/plugin-statefile.sh': No such file or directory" >&2
    echo "You must download PhrozenByte's statefile helper from https://daniel-rudolf.de/oss/munin-statefile-helper" >&2
    exit 1
fi

. "$MUNIN_LIBDIR/plugins/plugin.sh"
. "$MUNIN_LIBDIR/plugins/plugin-statefile.sh"

DF="${df:-df}"
MOUNTS="${mounts:-/proc/mounts}"

if [ "$1" == "autoconf" ]; then
    [ -x "$(which "$DF")" ] || { echo "no (\`$DF\` executable not found)"; exit 0; }
    [ -x "$(which "bc")" ] || { echo "no (\`bc\` executable not found)"; exit 0; }
    [ -f "$MOUNTS" ] || { echo "no ('$MOUNTS' not found)"; exit 0; }
    [ -r "$MOUNTS" ] || { echo "no ('$MOUNTS' not readable)"; exit 0; }

    echo "yes"
    exit 0
fi

if [ "$1" == "suggest" ]; then
    echo "pct"
    echo "abs"
    echo "inode_pct"
    echo "inode_abs"
    exit 0
fi

function contains {
    local LIST="$1"
    shift

    [ -n "$LIST" ] || return 1

    while (( $# > 0 )); do
        if [ -n "$1" ] && [[ " $LIST " == *" $1 "* ]]; then
            return 0
        fi
        shift
    done

    return 1
}

function matches {
    local REGEX="$1"
    shift

    [ -n "$REGEX" ] || return 1

    while (( $# > 0 )); do
        if [ -n "$1" ] && [[ "$1" =~ $REGEX ]]; then
            return 0
        fi
        shift
    done

    return 1
}

BASENAME="$(basename "$0")"

export warning="${warning:-92}"
export critical="${critical:-98}"

[ -n "$df_args" ] && readarray -t DF_ARGS <<< "$(xargs -n1 <<< "$df_args")" || DF_ARGS=()
DF_ARGS+=( -P )

EXCLUDE="${exclude:-none unknown rootfs iso9660 squashfs udf romfs ramfs tmpfs debugfs cgroup_root devtmpfs}"
EXCLUDE_REGEX="$exclude_re"
INCLUDE_REGEX="$include_re"

PERSISTENCE="${persistence:--6 hours}"
UNKNOWN_LIMIT="${unknown_limit:-3}"

PERCENT="yes"
if [ -z "$percent" ] || [ "$percent" == "auto" ]; then
    if [[ "$BASENAME" == *_abs ]]; then
        PERCENT="no"
    fi
elif [ "$percent" != "yes" ] && [ "$percent" != "on" ]; then
    PERCENT="no"
fi

INODES="no"
if [[ "$BASENAME" == *_inode ]] || [[ "$BASENAME" == *_inode_* ]]; then
    INODES="yes"
    DF_ARGS+=( -i )
elif [[ "${DF_ARGS[@]}" =~ (^| )(\-[a-zA-Z]*i[a-zA-Z]*|\-\-inodes)( |$) ]]; then
    INODES="yes"
fi

TOTAL="no"
if [ -z "$total" ] || [ "$total" == "auto" ]; then
    if [ "$PERCENT" != "yes" ] && [ "$INODES" != "yes" ]; then
        TOTAL="yes"
    fi
elif [ "$total" == "yes" ] || [ "$total" == "on" ]; then
    TOTAL="yes"
fi

DATA=()

while IFS=" " read -r DEV SIZE USED AVAIL _ MNT; do
    if ! matches "^[0-9]+$" "$SIZE" "$USED" "$AVAIL" || (( $SIZE == 0 )); then
        continue
    fi

    DEV_REGEX="$(echo "$DEV" | sed -e 's/[]\/$*.^[]/\\&/g')"
    MNT_REGEX="$(echo "$MNT" | sed -e 's/[]\/$*.^[]/\\&/g')"
    TYPE="$(sed -ne "s/^$DEV_REGEX $MNT_REGEX \([^ ]*\) .* [0-9]* [0-9]*$/\1/p" "$MOUNTS")"

    if contains "$EXCLUDE" "$TYPE" || matches "$EXCLUDE_REGEX" "$DEV" "$MNT"; then
        if ! matches "$INCLUDE_REGEX" "$DEV" "$MNT"; then
            continue
        fi
    fi

    ((SIZE = USED + AVAIL))

    FIELD="$(clean_fieldname "$DEV" "$MNT")"
    DATA+=( "$FIELD $DEV $MNT $SIZE $USED 0" )
    write_statefile "$FIELD" "DEV" "$DEV" "MNT" "$MNT" "SIZE" "$SIZE" "USED" "$USED" "GONE" "0" "LAST_SEEN" "$(date +%s)"
done < <("$DF" "${DF_ARGS[@]}" 2> /dev/null | tail -n +2)

while IFS= read -r FIELD; do
    if [ -n "$(printf '%s\n' "${DATA[@]}" | grep "^$FIELD ")" ]; then
        continue
    fi

    assign_statefile "$FIELD"
    if [ "$LAST_SEEN" -lt "$(date -d "$PERSISTENCE" +%s)" ]; then
        continue
    fi

    if [ "$1" != "config" ]; then
        ((++GONE))
    fi

    DATA+=( "$FIELD $DEV $MNT $SIZE $USED $GONE" )
    write_statefile "$FIELD" "DEV" "$DEV" "MNT" "$MNT" "SIZE" "$SIZE" "USED" "$USED" "GONE" "$GONE" "LAST_SEEN" "$LAST_SEEN"
done < <(read_statefile)

if [ "$1" == "config" ]; then
    if [ "$PERCENT" == "yes" ]; then
        if [ "$INODES" == "yes" ]; then
            echo "graph_title Inode usage (in percent)"
        else
            echo "graph_title Filesystem usage (in percent)"
        fi
        echo "graph_args --upper-limit 100 --lower-limit 0"
        echo "graph_scale no"
        echo "graph_vlabel %"
    elif [ "$INODES" == "yes" ]; then
        echo "graph_title Inode usage"
        echo "graph_args --base 1000 --lower-limit 0"
        echo "graph_vlabel inodes"
    else
        echo "graph_title Filesystem usage (in bytes)"
        echo "graph_args --base 1024 --lower-limit 0"
        echo "graph_vlabel bytes"
    fi
    echo "graph_category disk"
fi

TOTAL_SIZE=0
TOTAL_USED=0
for INDEX in "${!DATA[@]}"; do
    IFS=" " read -r FIELD DEV MNT SIZE USED GONE <<< "${DATA[$INDEX]}"

    if [ "$1" == "config" ]; then
        WARN="$(get_warning "$FIELD")"
        CRIT="$(get_critical "$FIELD")"

        echo "$FIELD.label $MNT"
        if [ "$PERCENT" == "yes" ]; then
            [ -z "$WARN" ] || echo "$FIELD.warning $WARN"
            [ -z "$CRIT" ] || echo "$FIELD.critical $CRIT"
        else
            [ "$INODES" == "yes" ] || echo "$FIELD.cdef $FIELD,1024,*"
            [ -z "$WARN" ] || echo "$FIELD.warning $((SIZE * WARN / 100))"
            [ -z "$CRIT" ] || echo "$FIELD.critical $((SIZE * CRIT / 100))"
        fi
        echo "$FIELD.unknown_limit $UNKNOWN_LIMIT"
    else
        if (( "$GONE" > 0 )); then
            echo "$FIELD.value U"
        elif [ "$PERCENT" == "yes" ]; then
            echo "$FIELD.value $(bc <<< "scale=2; 100 * $USED / $SIZE")"
        else
            echo "$FIELD.value $USED"
        fi

        if [ "$TOTAL" == "yes" ]; then
            ((TOTAL_SIZE += SIZE))
            ((TOTAL_USED += USED))
        fi
    fi
done

if [ "$TOTAL" == "yes" ]; then
    if [ "$1" == "config" ]; then
        echo "total.label Total"
        [ "$PERCENT" == "yes" ] || [ "$INODES" == "yes" ] || echo "total.cdef total,1024,*"
        echo "total.colour 000000"
    elif [ "$PERCENT" == "yes" ]; then
        echo "total.value $(bc <<< "scale=2; 100 * $TOTAL_USED / $TOTAL_SIZE")"
    else
        echo "total.value $TOTAL_USED"
    fi
fi

commit_statefile