Liveness and Readiness Checks for Splunk in K8S

splunk.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: splunk
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: splunk
    spec:
      containers:
        - name: splunk
          image: splunk/splunk:8.2
          imagePullPolicy: Always
          env:
            - name: SPLUNK_START_ARGS
              value: "--accept-license"
            - name: SPLUNK_PASSWORD
              value: "mypassword"

          # give splunk 5 minutes (30 * 10 = 300s) to startup
          startupProbe:
            exec:
              command:
                - /sbin/checkstate.sh
            failureThreshold: 30
            periodSeconds: 10

          # after startup, check liveness every 10 seconds
          livenessProbe:
            exec:
              command:
                - /sbin/checkstate.sh
            failureThreshold: 1
            periodSeconds: 10

          # don't send traffic to Splunk, unless it's ready (check every 5 seconds)
          readinessProbe:
            exec:
              command:
                - grep
                - started
                - /opt/container_artifact/splunk-container.state
            initialDelaySeconds: 10
            periodSeconds: 5

Usually, you'll use the Splunk Operator to run Splunk on K8S. There are some use cases where you might want to run Splunk without the operator though.

As with any deployment, it's good practice then to add liveness probes to restart Splunk if it's not healthy anymore.

Furthermore, as the Splunk container needs about one minute to startup, I'll recommend adding a readiness probe. This ensures that no traffic is sent to a pod as long as Splunk hasn't been fully started yet. Also, it won't terminate older instances on an update if the new instances are not ready yet.