Alertmanager test setup with basic auth for k8s-agent authentication testing

ALERTMANAGER_TEST_SETUP.md

Alertmanager Authentication Test Setup

This guide configures basic auth on your existing homelab Alertmanager for testing k8s-agent authentication.

Prerequisites

• Kubernetes cluster with Traefik ingress (Gateway API)
• Alertmanager already deployed at alertmanager.homelab.shindeiru.com
• Grafana available at grafana.homelab.shindeiru.com

Architecture

k8s-agent → Traefik (basic auth middleware) → Alertmanager ← Grafana (view alerts)

Step 1: Create Basic Auth Secret

# Generate htpasswd entry (admin:alertmanager-test)
# Using htpasswd:
htpasswd -nb admin alertmanager-test

# Or use this pre-generated hash for password "alertmanager-test":
# admin:$apr1$ruca84Hq$mbjdMZBAG.KWn7vfN/SNK/

Create the secret:

# alertmanager-basic-auth-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-basic-auth
  namespace: monitoring  # adjust to your Alertmanager namespace
type: Opaque
stringData:
  users: |
    admin:$apr1$ruca84Hq$mbjdMZBAG.KWn7vfN/SNK/

kubectl apply -f alertmanager-basic-auth-secret.yaml

Step 2: Create Traefik BasicAuth Middleware

# alertmanager-auth-middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: alertmanager-basic-auth
  namespace: monitoring  # adjust to your Alertmanager namespace
spec:
  basicAuth:
    secret: alertmanager-basic-auth
    removeHeader: true  # Don't pass auth header to backend

kubectl apply -f alertmanager-auth-middleware.yaml

Step 3: Update Alertmanager HTTPRoute

Add the middleware to your existing Alertmanager HTTPRoute or IngressRoute.

For Gateway API (HTTPRoute):

# Update your existing HTTPRoute to include the middleware annotation
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alertmanager
  namespace: monitoring
  annotations:
    traefik.io/middlewares: monitoring-alertmanager-basic-auth@kubernetescrd
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: traefik
  hostnames:
    - alertmanager.homelab.shindeiru.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: alertmanager
          port: 9093

For IngressRoute (Traefik CRD):

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: alertmanager
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`alertmanager.homelab.shindeiru.com`)
      kind: Rule
      middlewares:
        - name: alertmanager-basic-auth
      services:
        - name: alertmanager
          port: 9093
  tls:
    secretName: wildcard-homelab-cert  # your wildcard cert

kubectl apply -f <your-updated-route>.yaml

Step 4: Verify Authentication

# Should fail - 401 Unauthorized
curl https://alertmanager.homelab.shindeiru.com/api/v2/status

# Should work - with credentials
curl -u admin:alertmanager-test https://alertmanager.homelab.shindeiru.com/api/v2/status

Step 5: Configure k8s-agent

Option 1: Using .env file (recommended for local dev)

cp .env.example .env

Edit .env:

K8S_AGENT_ALERTMANAGER_AUTH_TYPE=basic
K8S_AGENT_ALERTMANAGER_AUTH_USERNAME=admin
K8S_AGENT_ALERTMANAGER_AUTH_PASSWORD=alertmanager-test
K8S_AGENT_ALERTMANAGER_URL=https://alertmanager.homelab.shindeiru.com

Run the agent:

source .env && ./bin/k8s-agent --config config.yaml

Option 2: Using password file

Create password file:

echo -n 'alertmanager-test' > /tmp/alertmanager-password

Update k8s-agent config:

# config.yaml
kubernetes:
  kubeconfig: ~/.kube/config
  namespaces:
    - default

alertmanager:
  url: https://alertmanager.homelab.shindeiru.com
  auth:
    type: basic
    username: admin
    password_file: /tmp/alertmanager-password

llm:
  model: your-model.gguf
  models_dir: ~/.lmstudio/models
  context_size: 4096

server:
  port: 8080
  log_level: debug

Run the agent:

./bin/k8s-agent --config config.yaml

Step 6: Test the Full Flow

1. Create a problem pod to trigger an alert:

   kubectl run crasher --image=busybox --restart=Always -- /bin/false

2. Watch k8s-agent logs for alert sending:

   # Look for "alert sent" or auth-related messages

3. View alerts in Grafana:

   • Open https://grafana.homelab.shindeiru.com
   • Go to Alerting → Alert rules
   • Or add Alertmanager as a data source if not already configured

4. View alerts in Alertmanager UI:

   • Open https://alertmanager.homelab.shindeiru.com
   • Login with admin:alertmanager-test

5. Clean up:

   kubectl delete pod crasher

Rollback

To remove authentication and restore open access:

# Remove middleware from HTTPRoute/IngressRoute
kubectl edit httproute alertmanager -n monitoring
# Remove the middleware annotation

# Or delete the middleware
kubectl delete middleware alertmanager-basic-auth -n monitoring
kubectl delete secret alertmanager-basic-auth -n monitoring

For K8s Deployment of k8s-agent

When deploying k8s-agent to the cluster (not running locally):

# k8s-agent secret
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-credentials
  namespace: default
type: Opaque
stringData:
  password: alertmanager-test

---
# k8s-agent deployment (partial)
spec:
  containers:
    - name: k8s-agent
      volumeMounts:
        - name: alertmanager-auth
          mountPath: /app/secrets/alertmanager
          readOnly: true
  volumes:
    - name: alertmanager-auth
      secret:
        secretName: alertmanager-credentials

Config would reference:

alertmanager:
  url: https://alertmanager.homelab.shindeiru.com
  auth:
    type: basic
    username: admin
    password_file: /app/secrets/alertmanager/password

Troubleshooting

401 after applying middleware

• Check middleware name matches in HTTPRoute annotation: namespace-middlewarename@kubernetescrd
• Verify secret exists: kubectl get secret alertmanager-basic-auth -n monitoring

Connection refused

• Verify Alertmanager service is running: kubectl get svc -n monitoring
• Check Traefik logs: kubectl logs -n traefik -l app.kubernetes.io/name=traefik

k8s-agent auth failures

• Test credentials manually first with curl
• Check password file has no trailing newline: cat -A /tmp/alertmanager-password
• Verify URL uses HTTPS (required for homelab with TLS)