Wowza JVM crash — run 9 analysis (2026-08-06)

wowza-jvm-crash-analysis-2026-08-06.md

Wowza JVM crash — run 9 + bb8e8c analysis (2026-08-06)

Summary

Two crashes are analysed here:

• Run bb8e8c — task bb8e8c0b7ed24fcabb243c3f5a77c015, crashed 2026-08-05 at 16:01:33 UTC, elapsed 29 minutes 48 seconds
• Run 3a84a76f (run 9) — task 3a84a76f896b4494a8fbfa1629142b28, crashed 2026-08-06 at 08:01:23 UTC, elapsed 12 minutes 20 seconds

Both crashes:

• Use the corrected 2304×1296 / 15 fps test input (confirmed from CloudWatch)
• Carry the identical crash signature as all previous runs (runs 1–8)
• Ran without -XX:-UseSHA because that flag has never been present in Parameter Store
• Refute the artefact document's conclusion that the crashes were "substantially a test artefact" — the crash still occurs on the corrected input; it is intermittent and correlates with churn rate (see section 5)

---

1. Crash comparison — all captured dumps

Field	bb8e8c (2026-08-05)	3a84a76f / run 9 (2026-08-06)
Signal	SIGSEGV SEGV_ACCERR	SIGSEGV SEGV_ACCERR
Fault address	0x00007f450ee4e000	0x00007f8057d8f000
Polling page	0x00007f450ee4e000 ✓	0x00007f8057d8f000 ✓
Fault addr == polling page	Yes	Yes
VM state	synchronizing (normal execution)	synchronizing (normal execution)
Thread identity	"Current thread is native thread"	"Current thread is native thread"
Faulting frame	java.util.TreeMap$NavigableSubMap$EntrySetView.size()	com.wowza.wms.livestreamrecord.model.LiveStreamRecorderBase$1.replace()
JIT tier	C2 (nmethod 12771)	C2 (nmethod 4918)
Elapsed at crash	29m 48s	12m 20s
Host CPU	Xeon Platinum 8175M (Sky Lake, 64 GB)	Xeon Platinum 8259CL (Cascade Lake, 32 GB)
Input	2304×1296, frameRate: 15.0 ✓	2304×1296, frameRate: 15.0 ✓
-XX:-UseSHA in JVM args	No	No
Heap used at crash	~3.8 GB of 20 GB	~770 MB of 20 GB

The faulting frame differs between runs (as with all previous dumps) — this is expected, since the safepoint poll is compiled into every method and the enclosing frame varies by thread scheduling at the moment the poll fires. The mechanism is identical in all cases.

---

2. -XX:-UseSHA — status and relevance

Both hs_err logs confirm the same 10-flag JVM arg list with no -XX:-UseSHA. Wowza's own startup log (CloudWatch, bb8e8c task, args 0–17) enumerates every flag explicitly — UseSHA is absent.

The current Parameter Store value (/wowza-streaming/common/JVM_EXTRA_ARGS, version 13, last modified 2026-08-06 08:48 UTC — after today's crash):

-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=30 -XX:MaxGCPauseMillis=200
-XX:+ExplicitGCInvokesConcurrent -XX:MaxMetaspaceSize=512m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/
-XX:OnOutOfMemoryError=/upload-heapdump.sh

entrypoint.sh sets -XX:-UseSHA in its default value for JVM_EXTRA_ARGS, but this default is only applied when JVM_EXTRA_ARGS is absent from the environment. The ECS task sources JVM_EXTRA_ARGS from Parameter Store, which overrides the entrypoint.sh default. As a result, bb8e8c and run 9 ran without the flag.

However, the flag is not a fix. It was present in prior run 8 (committed in e5f1069, deployed via the entrypoint.sh default when Parameter Store was not overriding), and run 8 crashed with the identical signature. -XX:-UseSHA disables SHA hardware intrinsics; sun.security.provider.SHA.implReset is not intrinsified by that flag, and the crash mechanism — safepoint poll / TLS identity loss — has no connection to SHA. Reintroducing it would have no effect and is not recommended.

---

3. The bb8e8c task — what the CloudWatch log tells us

Property	Value
Task start	2026-08-05 15:31:43 UTC
Crash time	2026-08-05 16:01:33 UTC (hs_err)
Elapsed	29m 48s
Input confirmed	frameSize: 2304x1296, frameRate: 15.0 (multiple streams)
Transcoded streams	511 unique, churned over a 7.3 m active window (~69.7 streams/min)
Stream IDs active near crash	wowza-churn-1785943912-005{08,09,10,...}
Crash exit	Aborted (core dumped), exit 134

The task's churn was concentrated in a ~7.3-minute active window at ~69.7 streams/min — well inside the crash band (see section 5). Elapsed time (29m 48s) is not the determining factor; churn rate is. Run 9 crashed at ~12 minutes; several 30-38 minute matrix runs at ≤ 18 streams/min did not crash.

---

4. Third distinct CPU microarchitecture

The three host CPUs seen across all captured crashes:

Task	CPU model	Microarch	RAM
Prior runs 1–8	Xeon Platinum 8375C	Ice Lake	8 GB
bb8e8c	Xeon Platinum 8175M	Sky Lake	64 GB
3a84a76f (run 9)	Xeon Platinum 8259CL	Cascade Lake	32 GB

The crash now reproduces on Sky Lake, Cascade Lake, and Ice Lake — three different CPU generations, two different RAM sizes. There is no host-specific or CPU-generation-specific component to this bug.

---

5. Crashes are churn-rate dependent — the matrix survivors under-tested

The matrix history is in the k6 repo

The survivor runs cited by the artefact document are recorded in projects/tv-link/wowza-only/wowza-churn-matrix-summary-2026-08-05.md. That summary confirms the survivors were genuinely on the corrected input (output_2304x1296_15fps_vbr_noise.mp4, 2304×1296 / 15 fps) — so the P2 question from the earlier draft is answered: yes, the matrix ran the corrected input. The mapping:

Matrix run	Task
wowza-4.11.0 @ concurrency 30	2a8693f2
wowza-4.11.1 @ concurrency 10	d519101e
wowza-4.11.1 @ concurrency 20	1af2a8fa

Crucially, the matrix only swept concurrency 5 → 10 → 20 → 30, and it was stopped manually at 13:41 UTC. Every crash discussed below occurred in a post-matrix run (after 13:41) at a churn rate higher than anything the matrix reached.

Churn rate is the discriminator

Measuring each loaded task's session-creation rate over its active churn window (first to last decoder-video-start, so survivors and crashers are compared on the same basis), all on the corrected input:

Task	Version / concurrency	Unique streams	Active churn window	Rate (streams/min)	Outcome
d519101e	4.11.1 / 10	208	32.2 m	6.5	Survived
6052ec39	post-matrix (config not recorded)	181	27.3 m	6.6	Survived
1af2a8fa	4.11.1 / 20	401	32.1 m	12.5	Survived
2a8693f2	4.11.0 / 30	204	11.2 m	18.2	Survived
3a84a76f (run 9)	post-matrix	513	11.9 m	43.0	Crashed
c29a4d39	post-matrix	450	9.2 m	49.2	Crashed
e946383e	post-matrix	413	6.6 m	62.4	Crashed
bb8e8c0b	post-matrix	511	7.3 m	69.7	Crashed
50d088f3	post-matrix	354	3.8 m	92.4	Crashed

("Unique streams" = distinct wowza-churn-* stream IDs with a decoder-video-start. "Rate" = unique streams ÷ active-window minutes: the average native transcoder-session creation rate during churn. For crashers the window ends at the crash.)

Key findings

1. A clean, wide separation at ~18 vs ~43 streams/min. Every survivor churned at ≤ 18.2 streams/min; every crasher at ≥ 43.0 streams/min. There is a ~25 streams/min gap with no overlap. The crash threshold lies somewhere in that gap.

2. The matrix survived because it never reached the crash rate. The matrix's maximum rung (concurrency 30, 2a8693f2) peaked at 18.2 streams/min — right at the top of the survivor band and well below the ≥ 43 streams/min crash band. The matrix did not demonstrate stability on the corrected input; it simply never pushed churn hard enough. The crashes all came later, at 2-5× that rate.

3. This is the strongest signal in the data and directly supports the root-cause hypothesis: crash probability scales with native transcoder session init/teardown rate (JNI attach/detach), which is exactly where the per-thread TLS disturbance that turns a recoverable safepoint-poll fault fatal would originate.

4. Elapsed time is not protective. bb8e8c crashed 31 minutes into its task; run 9 at ~12 minutes. What separates crashers from survivors is churn rate, not duration.

Corrections to earlier drafts

• An earlier draft reported c29a4d39/50d088f3 as lasting "~0.6 min" — taken mistakenly from the crash/upload tail window.
• A later draft computed churn rate as streams ÷ whole-task duration, which understated the crashers (e.g. bb8e8c at "16.5/min" — the churn was actually concentrated in a 7.3 m window at 69.7/min). All rates above use the active churn window consistently.
• The claim that the only credible survivor was a "6-stream" run was wrong; multiple 180-400-stream matrix runs survived.

The "590+ streams" figure

The artefact document attributes "590+ streams" to 2a8693f2. The task's transcoder actually started 204 unique streams (408 decoder-video-start events). The 590 figure most likely counts total ingest connection attempts (churn connect/disconnect cycles), not transcoded sessions. It does not change the conclusion: 2a8693f2 is a genuine corrected-input survivor — but at only 18.2 streams/min, below the crash band.

Bearing on the artefact document and matrix summary

Both documents observed real survivals on the corrected input, but both drew the wrong causal conclusion ("the corrected input fixes the crash"). The correct conclusion is that the corrected input was only ever tested at churn rates below the crash threshold. At ≥ 43 streams/min the crash reproduces reliably on the same corrected input.

---

6. Revised conclusions

What the artefact document got right

• The 640×400 / 90,000-fps malformed input is a genuine stress trigger and a separate robustness repro for Wowza — it crashes faster and more reliably.
• The corrected input (2304×1296 / 15 fps) is more representative of real device traffic.

What must be revised

1. The crash is not resolved. Two new crashes on the corrected input (bb8e8c at 29m 48s, run 9 at 12m 20s) with the identical signature refute "crashes were substantially a test artefact."

2. The crash reproduces on both inputs; it is not fixed by the corrected input. It occurs on both the malformed 90 kfps file and the corrected 2304×1296 / 15 fps file. On the corrected input it is churn-rate dependent: every survivor churned at ≤ 18.2 streams/min, every crasher at ≥ 43.0 streams/min (see section 5). The matrix that "proved" the input fix only ever reached 18.2 streams/min — below the crash band.

3. The crash is host- and CPU-independent. Three distinct CPU microarchitectures (Sky Lake / Cascade Lake / Ice Lake), two different RAM sizes, all crash identically.

4. The crash is JVM-flag-independent (in the flags tested so far). -XX:-UseSHA was present in run 8 and absent from bb8e8c and run 9 — all three crashed identically. The crash mechanism (safepoint poll / TLS identity loss) has no connection to SHA intrinsics. No JVM flag applied so far has had any effect.

5. Time-to-crash is governed by churn rate, not a fixed window. Run 9 crashed at 12m 20s; bb8e8c at 29m 48s; some 30-38 minute runs did not crash at all. What separates them is churn rate (crash threshold between ~18 and ~43 streams/min), not elapsed time.

Root cause (unchanged from prior analysis)

Recoverable JVM faults — specifically the safepoint poll (test [poll_page], SEGV_ACCERR at the mprotect-ed polling page) during VM state: synchronizing — are made fatal because the JVM signal handler reports "Current thread is native thread": Thread::current() / TLS / %fs cannot be resolved at signal time, so the handler cannot identify or recover the faulting thread and escalates to a fatal abort.

The faulting thread is a legitimate JavaThread (the poll pointer matches the real polling page; the crashing frames are ordinary Wowza worker code). The TLS identity loss is most plausibly caused by the native transcoding stack (libwms-transcoder.so, libmc_dec_avc.so, etc.) disturbing per-thread state at JNI attach/detach boundaries during high-rate session churn.

---

7. Immediate actions

Priority	Action
P0	Retract the artefact document's conclusion ("crashes were substantially a test artefact") — bb8e8c and run 9 both crash on the corrected input
P1	Update the Wowza support escalation doc to include bb8e8c and run 9, the third CPU microarchitecture data point, and the clarification that -XX:-UseSHA was tested in run 8 and did not prevent the crash
P1	Do not reintroduce -XX:-UseSHA — it was already tested in run 8 and has no effect on this bug
P1	Pin down the crash threshold with a controlled sweep on the corrected input across the ~18-43 streams/min gap (e.g. 20, 25, 30, 35, 40 streams/min held steady), rather than the concurrency-only matrix which topped out at 18.2 streams/min
P2	Once the threshold is known, treat it as an interim operational ceiling (cap the transcoder session-creation rate) while Wowza investigates the native root cause
P2	Correct wowza-churn-matrix-summary-2026-08-05.md and wowza-churn-input-artefact-2026-08-05.md: the survivors were real but only ran ≤ 18.2 streams/min; they did not test the crash-triggering churn band

---

6. Native component matrix + the 4.11.0 transcoder change (update 2026-08-06)

A cross-version audit of the native transcoder components (Wowza Streaming Engine 4.9.3 → 4.11.2, extracted from the official Docker images; 4.8.x is not published on Docker Hub) resolves which native code moved into the crash window.

What is NOT the cause. Every MainConcept codec library (libmc_dec_*, libmc_enc_*, libmc_trans_*) is byte-identical (same sha256) from 4.9.7 through 4.11.2. The HW-accel back-ends (NVENC, Xilinx U30, QuickSync/oneVPL), libvpx, ogg/ vorbis, libstdc++ and Intel OpenMP are also unchanged across that span. The crash reproduces on 4.11.0/4.11.1, so it cannot be a MainConcept codec regression.

The one native component that changes in the crash window is Wowza's own libwms-transcoder.so (itself byte-identical across 4.11.0/4.11.1/4.11.2). The last native change before the crashers is therefore 4.10.0 → 4.11.0.

What changed in 4.11.0 (from disassembly of the non-stripped .symtab):

• A new JNI method TranscoderSession::videoDecoderReleaseDecodedFrame(uint) that, on a JVM-owned thread, takes a std::mutex and calls condition_variable::notify_one() for every decoded frame.
• TranscoderSession::resetPipeline() grew 446 → 516 bytes: 4.11.0 appended a lock_guard<std::mutex> + condition_variable::notify_all(), run once per session reset/teardown. (~TranscoderSession is unchanged.)
• Both use the same member mutex/condition_variable (TranscoderSession + 0x998). This handshake did not exist in 4.10.0.

Why this matters. 4.11.0 introduced a producer/consumer signalling mechanism in which JVM threads hold a native mutex and wake native decoder pthreads on paths whose frequency scales directly with transcoder-session churn — notify_all per session reset, notify_one per decoded frame. That is exactly the JVM-thread-in-native- synchronisation pattern that can extend the window in which a JVM thread is mid-native- call when a global safepoint is requested, which is consistent with the hs_err "native thread" + safepoint polling-page signature and with the ≥ 43 streams/min crash band in section 5.

Probable cause. The SIGSEGV on the safepoint polling page is most likely triggered by the mutex/condition-variable handshake Wowza added to libwms-transcoder.so in 4.11.0. Under high session churn, JVM-owned threads repeatedly enter native code in TranscoderSession (resetPipeline → notify_all per teardown, videoDecoderReleaseDecodedFrame → notify_one per frame), where they take a native std::mutex and block/wake on the condition variable. This lengthens and multiplies the windows in which a JVM thread is parked inside native pthread synchronisation at the moment a global safepoint is requested; when the JVM arms the polling page and the thread's safepoint state / TLS is disturbed by that native blocking, the normally recoverable poll fault becomes fatal — producing the observed "Current thread is native thread" + polling-page SEGV_ACCERR signature. The crash is therefore rate-driven (a concurrency/safepoint interaction that scales with churn), not a data-corruption bug in any codec — consistent with the clean survivor/crasher split at ~18 vs ~43 streams/min and with the crash reproducing across three CPU microarchitectures.

Status: this is a correlation from static analysis, not proof of causation — but it is the only semantic native change between the low-churn survivors and the reliable crashers, so it is the highest-value target for dynamic confirmation.

Recommended actions (additions)

Priority	Action
P1	Dynamically confirm/refute the CV hypothesis: reproduce at ≥ 43 streams/min and trace condition_variable wakeup rate (bpftrace/perf uprobes on the two TranscoderSession symbols on an EC2 host, or an LD_PRELOAD shim on pthread_cond_signal/broadcast if repro is Fargate-only). Confirmation = wakeup rate crosses into the crash band before every crash and stays low for survivors.
P2	Raise the 4.10.0 → 4.11.0 TranscoderSession mutex/CV change with Wowza support as the specific suspected native regression, alongside the crash threshold sweep.
P2	As an interim, evaluate pinning to a pre-4.11.0 engine (native transcoder codecs are unchanged since 4.9.7, so functionally equivalent) if the crash threshold cannot be kept above the operational churn ceiling.

---

Addendum (2026-08-17): resolved in 4.11.3 — and the root cause was different

Wowza 4.11.3 resolves this crash: a 16 vCPU / 32 GB ECS task now sustains ~360 concurrent streams. A binary diff of 4.11.2 → 4.11.3 (linux/amd64) plus NFT validation is written up in wowza-4.11.3-fix-analysis-2026-08-17.md. Key points that revise this document:

• The fix is native, in libwms-transcoder.so — not in any JAR. Every wms-*.jar is rebuilt but only by rebuild-noise amounts; the one targeted change is native.
• The actual root cause is glibc malloc-arena growth under churn, not the 4.11.0 mutex/CV handshake. 4.11.3 adds a malloc_trim@GLIBC_2.2.5 import (absent in 4.11.2) and calls it (rate-limited) on TranscoderSessionNative.destroyInstance, plus warns if MALLOC_ARENA_MAX is unset. The resetPipeline / videoDecoderReleaseDecodedFrame functions flagged in §6 above (and in native-component-matrix.md) are unchanged in 4.11.3 (identical sizes), so that mutex/CV hypothesis is superseded — it was a correlation, not the defect.
• NFT confirms the fix is live: 4.11.3+11 ran on Aug 14 at ~57k churn events with the new destroyInstance ... returned free native heap pages to the OS (1–5ms) message firing and the MALLOC_ARENA_MAX warning silent (arenas capped).
• Memory: at NFT load the total footprint is dominated by the 20 GB JVM heap (-Xmx20000M); the native arenas the fix tunes are negligible there, so there is no measurable memory change up or down at NFT scale. The fix matters at the ≥43 streams/min band on memory-constrained tasks — exactly the regime that produced the crashes in §5.