performance-k8s-container.md

On Kubernetes, the biggest wins usually come from: right-sizing CPU/memory to avoid throttling, matching app concurrency to DB/Redis limits, and separately tuning Puma (web) vs Sidekiq (jobs) so they don’t fight for the same resources.

Below is a practical tuning playbook you can apply immediately.

---

1) First: avoid the classic k8s performance trap (CPU throttling)

If your pods have a low CPU limit, the Linux CFS scheduler will throttle Ruby, and p95/p99 latency goes bad.

• For web pods, strongly consider:

  • set CPU requests, and either no CPU limit or a high enough limit that you don’t throttle under normal load.

• For Sidekiq pods, CPU limits can also throttle and increase job latency.

Quick check:

• Look at container_cpu_cfs_throttled_seconds_total and request latency spikes.

Rule of thumb:

• Web pods want predictable CPU, sidekiq wants enough CPU per concurrency.

---

2) Separate web and sidekiq into different Deployments

If they share a pod, they’ll contend for CPU, memory, and GC time. You want independent:

• resource requests/limits
• HPA scaling rules
• rollouts
• pod disruption budgets

---

3) Puma tuning for Rails web server

Pick workers/threads based on CPU + DB pool

• Workers are OS processes (memory heavy, good for CPU parallelism).
• Threads are concurrency inside a process (good for I/O wait, but needs DB pool headroom).

Good starting point in containers:

If you have 2 vCPU per pod

• WEB_CONCURRENCY (workers): 2
• RAILS_MAX_THREADS (threads): 5 (or 3–5)

If you have 1 vCPU per pod

• workers: 1
• threads: 5

Then set DB pool:

• Rails pool should be at least: workers * threads + a small buffer
• Example: 2 workers × 5 threads = 10 → set pool: 12 (and ensure Postgres max connections can handle total pods)

Preload app

Enable preload_app! so copy-on-write reduces memory per worker (usually helps a lot in containers).

Kill slow requests (protect tail latency)

Set:

• worker_timeout (Puma) reasonable for your endpoints
• app-level timeouts for external calls (HTTP, etc.)

Use multiple replicas + HPA

Scale on:

• CPU and/or request rate (if you have metrics)
• Keep enough replicas to survive node drains without big latency spikes.

---

4) Sidekiq tuning (jobs)

Sidekiq throughput is mostly concurrency × CPU × DB/Redis capacity, plus job type.

Start with conservative concurrency per CPU

For typical Rails jobs:

• Concurrency ~ 5–10 per vCPU is a common starting range.
• If jobs are DB-heavy, start lower (e.g., 5).
• If jobs are mostly network I/O, you can go higher.

Example starting points:

• 1 vCPU Sidekiq pod → -c 10 (or 5 if DB-heavy)
• 2 vCPU → -c 20 (or 10 if DB-heavy)

Use multiple queues with weights (prevent starvation)

Set queues like:

• critical, default, low and weight critical higher so long low-priority jobs don’t block important work.

Scale Sidekiq based on queue depth

Use KEDA (common in k8s) to scale Sidekiq replicas off Redis queue length.

• This is usually better than CPU-based scaling for jobs.

Ensure DB pool matches Sidekiq concurrency

Sidekiq uses ActiveRecord connections too. Set DB pool in Sidekiq deployment to at least:

• SIDEKIQ_CONCURRENCY + small buffer

If web and sidekiq share the same DB, total connections = (web pods × web per-pod connections) + (sidekiq pods × sidekiq per-pod connections). This is where systems often fall over.

---

5) Ruby/GC/memory tuning in containers

YJIT (if you’re on Ruby 3.2+)

If your service is CPU-bound, YJIT often helps web throughput. Try enabling it on web pods first:

• RUBY_YJIT_ENABLE=1 (or start with ruby --yjit) Watch CPU and memory—YJIT uses extra memory.

GC settings (especially important for Sidekiq)

Ruby GC tuning can reduce latency spikes:

• Ensure you’re not hitting memory limit → OOMKills
• Give Ruby enough headroom; GC behaves badly when memory is tight.

Also consider using jemalloc (many Ruby services see less fragmentation):

• preload jemalloc in the container and compare RSS + p99.

---

6) Database and Redis: the real bottlenecks

Web + Sidekiq performance usually ends up limited by:

• Postgres connection count / slow queries
• Redis latency / saturation
• External APIs

Checklist:

• Add/confirm indexes for top queries and job queries
• Reduce N+1s
• Set statement timeouts for safety
• Ensure Redis is not CPU-starved (especially if it’s also in k8s)

---

7) Deployment-level knobs that matter

• Readiness probe: only mark ready once app is actually ready (DB, migrations strategy, etc.)

• Graceful shutdown:

  • Puma: enough terminationGracePeriodSeconds to finish in-flight
  • Sidekiq: quiet + stop so jobs aren’t killed mid-run

• PodDisruptionBudget: avoid dropping too many web replicas at once

---

8) What to measure (so tuning isn’t guesswork)

Minimum dashboards:

• Web: RPS, p50/p95/p99, error rate, CPU throttling, RSS, GC time
• Sidekiq: queue latency, processed/sec, retries/dead, CPU throttling, RSS, GC time
• DB: connections, slow queries, lock time
• Redis: ops/sec, latency, CPU

---

Quick “good defaults” to try (as a baseline)

Web pod (2 vCPU / 2–4Gi RAM)

• Puma workers: 2
• Threads: 5
• Rails DB pool: 12
• Enable preload_app
• Consider YJIT

Sidekiq pod (2 vCPU / 2–4Gi RAM)

• Concurrency: 10–20 depending on DB heaviness
• DB pool: concurrency + 2
• KEDA scaling by queue depth