Skip to content

Instantly share code, notes, and snippets.

@seupedro

seupedro/REPORT.md

Last active April 24, 2026 12:11
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save seupedro/67719195d2eef076a2a87aac65eac061 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save seupedro/67719195d2eef076a2a87aac65eac061 to your computer and use it in GitHub Desktop.
Download ZIP
SongsForge v0.2.3 100-job stress test — 2026-04-24
Raw
REPORT.md

SongsForge 100-job stress test — v0.2.3-stack

Host: 4 vCPU AMD EPYC-Genoa / 7.6 GiB / 0 swap — api.songsforge.dev Window: 2026-04-24 09:06:24 → 10:29:20 UTC (≈ 83 min) Monitor samples: 1 203 host rows + 10 828 per-container rows (every 2 s)

Result at a glance
jobs fired in parallel 100 (30 certificate + 30 pdf + 30 ebook + 10 karaoke with 30 Wikimedia photos each)
artifacts produced 99 / 100
Hatchet succeeded 99 / 100 (99.0 %)
failures 1 ebook — upstream OpenAI chat-completion timeout (external service, not our stack)
scheduleTimeout cancellations (old v0.1.0 failure mode) 0 ✅
gateway 400s (bad payload rejected sync) 0 ✅
cert executionTimeout failures (v0.2.2 edge case fixed in v0.2.3) 0 ✅
OOM kills 0 ✅

Host peaks

Metric Baseline Peak
load1 0.20 4.62 (≈ 115 % of 4 cores, instantaneous)
load5 0.38 2.01
load15 0.58 1.56
mem used (MB / 7 752) 2 505 4 387 (56.6 %)
mem free (MB) 622 139 (min)

Container peaks

Container peak CPU peak mem % peak mem
karaoke-worker 106.85 % 52.04 % 1.041 GiB / 2 GiB
karaoke-whisper 84.79 % 65.38 % 669 MiB / 1 GiB
postgres 80.65 % 78.84 % 1.183 GiB / 1.5 GiB
pdf-worker 79.65 % 30.26 % 310 MiB / 1 GiB
gateway 40.41 % 11.03 % 56 MiB / 512 MiB
ebook-worker 33.31 % 65.64 % 252 MiB / 384 MiB
certificate-worker 32.97 % 70.65 % 361 MiB / 512 MiB
hatchet-lite 21.08 % 9.20 % 94 MiB / 1 GiB
caddy 6.74 % 36.24 % 46 MiB / 128 MiB

Highlights

  • karaoke-worker peaked at 106.85 % — the cpus: 1.0 cgroup cap is holding (docker stats sees brief sampling overshoot; kernel enforces 100 % over any averaged window).
  • certificate-worker peaked at 70.65 % of the new 512 MiB cap — the v0.2.3 bump from 384 MiB (which hit 94.2 % in the prior 47-job spike) did its job. Zero cert failures this run.
  • postgres peaked at 78.84 % of 1.5 GiB — fine, but a 200-job spike would want 2 GiB.
  • gateway at 40.41 % CPU was the first time it has been meaningfully loaded. 100 parallel POSTs + zod payload validation absorbed into its 0.5 CPU cap.

Timeline (drain)

App Count Drain time Notes
certificate 30 ~ 5 min SLOTS=10, fast pdf-lib renders
ebook 30 ~ 3 min SLOTS=10, concurrent OpenAI calls
pdf 30 ~ 25 min SLOTS=4, Kie-bound
karaoke 10 ~ 75 min SLOTS=1 serial, 30-photo xfade montage per video

Changes since v0.2.2

  1. certificate-worker mem_limit: 384m → 512m (docker-compose.yml).
  2. certificate.generate executionTimeout: "2m" → "5m" (apps/certificate/src/main.ts).

Both findings surfaced by the prior 47-job spike. Zero cert failures this run — fixes validated.

One thing worth flagging

The only caller-visible failure was an OpenAI timeout on one ebook. All workers run retries: 0 — any transient external blip surfaces as a failed job. If you want this hidden from callers, set retries: 1 on ebook + pdf (both OpenAI-dependent) for a single free retry with backoff.

certificate — 30/30

pdf — 30/30

ebook — 29/30

karaoke — 10/10

Methodology

  • 100 POSTs fired in parallel via bash (one & per request), all from outside the VPS hitting https://api.songsforge.dev.
  • Each karaoke request supplies an albumId referring to a 30-photo album previously uploaded to POST /karaoke/photos. 4 unique albums were cycled across the 10 karaoke requests.
  • VPS resource monitor (/root/vps-monitor.sh) sampled /proc/loadavg, /proc/meminfo, df /, and docker stats --no-stream every 2 s.
  • Polling loop hit GET /:app/:jobId every 15 s until every job reached a terminal status.

Conclusion

v0.2.3 absorbed a 100-job parallel burst (2× any prior test) with 99.0 % success and no stack-side failures. The sole failure was an external OpenAI timeout on one ebook — a retry path the workers do not currently exercise. Host resources held comfortably within caps throughout the 83-minute window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment