RTSP Debug

rtsp-debug-context.md

RTSP/HEVC Ingest Debugging — Handoff for Scout

Situation

• End customer's video feed plays cleanly in VLC.
• Same RTSP URL fed into Scout's Media Server pipeline produces visible video artifacts.
• Source chain: IP camera → exacqVision NVR → RTSP republish → Scout's Media Server.
• Stream is HEVC, 2048x1536, 10 fps, yuvj420p, advertised SDP title H265, streamed by rtspserverpi, RTSP path /3480576.
• Diagnostic was run on Scout's own machine: C:\Program Files\RunScoutAI\Media Server\ffmpeg.exe (ffmpeg 8.0-essentials build).
• Artifact pattern is identical over both -rtsp_transport tcp and -rtsp_transport udp. Network packet loss is therefore not the root cause.

Diagnostic output (relevant excerpts)

Stream identity (both runs)

Input #0, rtsp, from 'rtsp://rtspuser:****@10.110.116.4:8554/3480576':
  Metadata:
    title           : H265, streamed by rtspserverpi
    comment         : 3480576
  Duration: N/A, start: 387.097000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc, bt709), 2048x1536 [SAR 1:1 DAR 4:3], 10 fps

Startup errors (every connection)

[hevc] PPS id out of range: 0
[hevc] Skipping invalid undecodable NALU: 1
... repeats many times ...

Mid-stream errors (sequential POCs)

[hevc] Could not find ref with POC 28
[hevc] Error constructing the frame RPS.
[hevc] Skipping invalid undecodable NALU: 1
[hevc] First slice in a frame missing.
... POC 29, 30, 31, 32, ... all the way through 48 ...

Muxer warning

[null] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 176 >= 176

Final outcome (TCP run)

frame=  717 fps= 10 q=-0.0 size=N/A time=00:01:13.70 bitrate=N/A speed=1.03x elapsed=0:01:11.87

717 frames decoded over ~74 seconds at the source's 10 fps cadence — frames are coming through, but startup and periodic stretches are corrupt.

What the errors mean

PPS id out of range: 0 + Skipping invalid undecodable NALU: 1

NALU type 1 in HEVC is TRAIL_N (a non-IDR inter-frame slice). The decoder is receiving slice NALUs before it has the matching VPS / SPS / PPS parameter sets. Every TRAIL_N gets dropped until the first IDR (with embedded headers) shows up.

Two source-side causes for this normally:

• Encoder doesn't repeat VPS/SPS/PPS in-band before each IDR.
• The SDP sprop-vps / sprop-sps / sprop-pps advertised in DESCRIBE is missing or malformed, so the client never seeds the decoder out-of-band.

But VLC plays the same URL cleanly. So whatever exacq is sending is parseable — VLC's live555 stack handles it. The failure is in how Scout's client (ffmpeg 8.0 in the diagnostic) ingests it.

Could not find ref with POC N / Error constructing the frame RPS / First slice in a frame missing

POC = Picture Order Count. RPS = Reference Picture Set. The decoder is being told "this P/B slice references frame POC=N", but POC=N was never decoded (it was dropped earlier as part of the parameter-set startup problem, not because of network loss).

POCs march sequentially (28, 29, 30, …, 48), one per frame. That is not bursty packet loss. That is downstream fallout from the startup parameter-set miss: the decoder never recovered cleanly.

non monotonically increasing dts (176 >= 176)

The pipeline's timestamp generator is producing duplicate DTS values. Probably a side effect of the reference-frame errors above (slices being processed twice or being emitted with the same timestamp).

Why TCP and UDP fail identically

TCP has zero packet loss. If the artifacts persist over RTSP/TCP, the issue is not in the network path between exacq and Scout. It's in client-side parsing/decoding inside Scout's Media Server.

Reproduction scope tightens the diagnosis

Scout has confirmed: this only happens on feeds coming from this exacq NVR. Other RTSP sources ingest cleanly into Scout's pipeline.

That eliminates "generic ffmpeg HEVC-over-RTSP regression" as the lone explanation. ffmpeg 8.0 might still be a contributing factor, but the bug is specifically in how Scout's RTSP client interacts with what exacq sends. Things exacq's RTSP republish layer is known to do that can confuse non-live555 clients:

• Lazy keyframe-on-join: exacq doesn't always trigger an immediate IDR when a new RTSP client connects. The client must either wait for the next scheduled IDR (could be many seconds depending on the camera's GOP length) or rely on SDP-advertised parameter sets to seed the decoder. VLC/live555 handles this. ffmpeg's RTSP module historically has not handled it as gracefully.
• SDP formatting quirks: exacq's SDP for HEVC sometimes formats sprop-vps/sps/pps in a way that older or newer ffmpeg builds parse differently. live555 is more forgiving.
• RTP packetization: HEVC over RTP (RFC 7798) supports Single NAL units, Aggregation Packets, and Fragmentation Units. Exacq's choice of packetization mode and how it splits large NALUs may not match what ffmpeg's HEVC depacketizer expects.
• SETUP transport negotiation: exacq may handle TCP-interleaved RTSP differently than UDP, but since both fail identically here, that's likely not the diff.

The fact that the failure is exacq-specific means tests #4 (capture and inspect SDP), #6 (compare VLC vs ffmpeg negotiation), and #7 (Wireshark diff) are now the highest-value diagnostics. The diff between what VLC sees and what ffmpeg sees on the same exacq stream is the answer.

Most likely root causes (in priority order)

1. ffmpeg 8.0 HEVC-over-RTSP regression

ffmpeg 8.0 is a very new release. There have been HEVC/RTSP regressions in recent ffmpeg builds. VLC works because it uses live555 internally, not libavformat. Worth testing ffmpeg 7.1 LTS or 6.1 LTS as a known-good baseline.

2. SDP parameter-set parsing

A correct RTSP client base64-decodes sprop-vps / sprop-sps / sprop-pps from the SDP and feeds them to the HEVC decoder before any RTP packet arrives. If Scout's client skips that (or has a bug parsing exacq's specific SDP), the decoder waits for in-band parameter sets, which only show up at the next IDR. Everything before that IDR is dropped.

This matches the symptom exactly. VLC primes its decoder from the SDP (via live555); ffmpeg's RTSP module may not be handling exacq's SDP the same way.

3. Production pipeline ≠ diagnostic ffmpeg

The ffmpeg CLI run in C:\Program Files\RunScoutAI\Media Server may not be what actually decodes video in production. Production may use an embedded library (OpenCV, GStreamer, live555, a managed wrapper, custom code). The CLI repro is useful but not necessarily the bug location.

Diagnostic tests for Scout to run

Pin down the production decoder

1. In production, what is actually pulling and decoding the RTSP feed inside Scout's Media Server? CLI ffmpeg, libav bindings, OpenCV, GStreamer, live555, a managed wrapper, custom code?
2. Is the version of that library/binary the same as the diagnostic (ffmpeg 8.0)?

Quick wins

3. Downgrade CLI ffmpeg to 7.1 LTS or 6.1 LTS. Rerun the same -rtsp_transport tcp -i ... -f null - test. Compare the error pattern.
4. Confirm exacq's SDP advertises parameter sets:

   ffmpeg -rtsp_transport tcp -loglevel trace -i "rtsp://..." -frames:v 1 -f null - 2>&1 | findstr /i "sprop sdp= a=fmtp"
   

   We want to see sprop-vps=..., sprop-sps=..., sprop-pps=.... If present, exacq is doing its job; failure is on Scout's side.

Record-and-replay split test

5. Capture the raw stream with -c copy (no re-decode):

   ffmpeg -rtsp_transport tcp -i "rtsp://..." -c copy -t 30 raw.mp4
   

   Then open raw.mp4 in VLC.
   • VLC plays raw.mp4 cleanly → ffmpeg's RTSP demux is fine; the artifacts come from the decode/display pipeline downstream.
   • VLC shows the same artifacts in raw.mp4 → ffmpeg is mangling the bitstream during RTSP ingest itself.

Side-by-side compare with VLC

6. In VLC, while playing the stream, open Tools → Codec Information and screenshot it. We want the negotiated transport (TCP/UDP), codec, and parameters. Compare against what Scout's pipeline negotiates.
7. Wireshark capture of 30 seconds of stream traffic during a VLC session and a Scout session, both filtered to the camera/NVR IP. Diff the DESCRIBE response and the first ~50 RTP packets. If the SDP is identical, the difference is purely client-side handling.

Reproduction scope

8. Confirmed: this ONLY happens on feeds coming from this exacq NVR. Other RTSP sources ingest cleanly into Scout's pipeline. See "Reproduction scope tightens the diagnosis" below.
9. If exacq is reconfigured to publish this channel as H.264 instead of HEVC, does the artifact go away in Scout? HEVC support in RTSP-ingest libraries is less mature than H.264. Switching the exacq channel to H.264 may be the fastest unblock for the end customer while Scout fixes the underlying compatibility issue.

Useful ground truth already established

• Bitstream from exacq is good (VLC plays cleanly).
• TCP and UDP fail the same way (rules out network).
• ffmpeg 8.0 on Scout's machine fails at parameter-set seeding.
• Decoder eventually catches up after the first IDR, but the startup and POC-miss windows produce visible corruption.

Suggested next actions (revised given exacq-only scope)

1. Wireshark diff (test #7). Capture 30 seconds of RTSP+RTP traffic for VLC pulling the exacq stream, then 30 seconds of Scout's pipeline pulling the same stream. Diff the DESCRIBE response (SDP) and the first ~50 RTP packets. The diff between "what VLC sees and decodes cleanly" vs "what Scout sees and breaks on" is the answer. This is the single highest-value test now that the bug is known to be exacq-specific.
2. Record-and-replay split (test #5). 30-second test that decisively separates "ingest is broken" from "decode is broken." Cheap and informative.
3. H.264 fallback (test #9). Quickest unblock for the end customer. If exacq can republish this channel as H.264, Scout's pipeline may handle it cleanly even while the HEVC compatibility issue remains open.
4. ffmpeg version sweep (test #3). If Scout's production decoder is libav-based, try 7.1 LTS and 6.1 LTS against this exacq stream specifically. Compare error patterns.

The Wireshark diff is the one that will actually identify the root cause. Everything else is a workaround or a narrowing aid.