Skip to content

Instantly share code, notes, and snippets.

@denislemire

denislemire/incident-20260305-plex.md

Created March 5, 2026 18:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denislemire/228537c98f2f403fd015e31ad7b55259 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denislemire/228537c98f2f403fd015e31ad7b55259 to your computer and use it in GitHub Desktop.
Download ZIP
Incident 2026-03-05: Plex 502/404 on EHWS cluster
Raw
incident-20260305-plex.md

Incident Report: Plex 502 / 404 on EHWS Cluster

  • Date: March 5, 2026
  • Service: Plex Media Server — https://plex.ehws.generic.business
  • Status: Resolved
  • Impact: Plex unreachable (502 Bad Gateway, then 404) for internal/VPN users and via inferno proxy

Summary

Plex on the EHWS Kubernetes cluster was returning 502 Bad Gateway (later 404) when accessed at https://plex.ehws.generic.business. Traffic flows from clients to Traefik (192.168.12.102) via inferno nginx or AdGuard DNS rewrites. Two separate issues were identified and fixed:

  1. Ingress: No Kubernetes Ingress existed for Plex; the host was later added but with the wrong backend service name and missing Traefik annotations, so Traefik either had no route (502) or routed to a non-existent service (404).
  2. Plex process not starting: The Helm release forced a non-root securityContext; the Plex image (plexinc/pms-docker) uses s6-overlay and fails with s6-setuidgid: Permission denied when run as UID 1000, so no process listened on port 32400 (connection refused → bad gateway).

Timeline

Phase Symptom Finding
Initial 502 Bad Gateway Traffic to plex.ehws.generic.business goes to Traefik; no Ingress for that host. Plex was deployed with ingress.enabled: false and LoadBalancer only.
After adding Ingress 502 / Bad Gateway Ingress backend was set to service name plex; the Helm chart creates plex-plex-media-server. Traefik could not find backend.
After fixing backend name Bad Gateway Traefik was using HTTPS to talk to backend; Plex serves HTTP on 32400. Needed serversscheme: http. Also needed router.entrypoints: websecure for HTTPS.
After annotations Bad Gateway Backend was reachable from Traefik but connection refused: Plex pod was not listening on 32400.
Log inspection — Container logs: s6-setuidgid: Permission denied in a loop; Plex process never started.
After removing securityContext 404 (post-merge) Flux reverted Ingress to main (wrong backend plex again). Applied correct Ingress from repo; created new PR so fix persists.

Root Causes

1. Ingress configuration

  • Backend service name: Ingress must point to plex-plex-media-server (Helm releaseName + chart app name), not plex.
  • Traefik entrypoint: Requests arrive on HTTPS (inferno/AdGuard → Traefik:443). The Ingress must be bound to the websecure entrypoint via annotation: traefik.ingress.kubernetes.io/router.entrypoints: websecure.
  • Backend protocol: Plex listens on HTTP on port 32400. Traefik must use HTTP to the backend: traefik.ingress.kubernetes.io/service.serversscheme: http.

2. Plex container not starting

  • securityContext: The Helm values set pms.securityContext: { runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 }.
  • Image behavior: plexinc/pms-docker uses s6-overlay; the entrypoint runs /usr/bin/s6-setuidgid, which fails with "Permission denied" when the container is already running as non-root.
  • Effect: The main Plex process never starts; nothing listens on 32400 → connection refused from Traefik → Bad Gateway.

Resolution

Ingress (clusters/ehws/plex/ingress.yaml)

  • Backend service: plex-plex-media-server, port 32400.
  • Annotations:
    • cert-manager.io/cluster-issuer: letsencrypt-dns
    • traefik.ingress.kubernetes.io/router.entrypoints: websecure
    • traefik.ingress.kubernetes.io/service.serversscheme: http

Helm release (clusters/ehws/plex/helm-release.yaml)

  • Removed pms.securityContext (runAsUser, runAsGroup, fsGroup). Left a comment that the default (empty) is required so s6-overlay can start.

Cluster state (one-time)

  • Patched the StatefulSet to remove container securityContext and restarted the Plex pod so it could start. After merging the PR, Flux keeps the correct Ingress and Helm values.

Verification

  • From inside the cluster: curl -s -o /dev/null -w "%{http_code}" http://plex-plex-media-server.plex.svc.cluster.local:32400/web → 200 (or 302).
  • Ingress in cluster: kubectl get ingress plex -n plex shows ADDRESS 192.168.12.102; backend and annotations correct.
  • Browser: https://plex.ehws.generic.business loads Plex (or setup).

References

  • ehws-infra: clusters/ehws/plex/ — Ingress, HelmRelease, kustomization.
  • PR: fix(plex): correct Ingress backend and allow Plex to start (fix/plex-ingress-and-startup).
  • AdGuard rewrites and inferno nginx already pointed plex.ehws.generic.business to Traefik; no change there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment