AV1 Overlapping bitrate session summary

av1-overlapping-bitrates-summary.md

Debug session: AV1 retranscoding with overlapping bitrate

Date: 2026-09-22 Branch: bugfix/overlap-av1-bitrate Base commit: df5879cd2 (fix error in test, correct smil content)

The task

A failing test existed in tests/IsolatedTests/PsTests.fs:

When a program gets retranscoding with disabled codec and overlapping bitrate, s3 contains files and smil includes latest fileset

The test was written first (commits 83e9ba20d, df5879cd2) to reproduce a suspected business-logic bug. The task was to locate the bug in the production code so the test passes.

What the test does

1. Publishes a program with 5 h264 files at transcoding version 170000, bit rates 1..5. Waits for 5 mp4s + 1 smil in S3.
2. Publishes a second transcoding at version 170001 with 5 new files — the first of which (bit rate 1) is AV1, the rest h264.
3. Asserts S3 holds 5 + 5 - 1 = 9 mp4s (the AV1 file is never uploaded) and 2 smils.
4. Asserts the latest smil lists exactly the 4 new non-AV1 files.

Step 4 was the failing assertion.

Investigation path

• grep -i av1 over the repo → three production hits:
  • src/Core/Config.fs:570 — the enableAV1Upload feature-flag key
  • src/Core/Domain/ActionContext.fs:32,61 — EnableAV1Upload on the action context
  • src/Core/Domain/GlobalConnectActions.fs:227 — the actual gate
• isContentSupportedByOrigin (GlobalConnectActions.fs:225) is used in exactly two places: buildUploadFileActions (line 240) and getActiveFiles (line 509). Notably not in the smil construction.
• Followed the smil path: buildSmilActions → GlobalConnectSmil.fromCurrentState (src/Core/Domain/MediaSetTypes.fs:1323), also reached from GlobalConnectFileUploadActor.fs:223.
• Checked how FileRef / QualityId are derived: PsJobCreation.fs:366 builds QualityId.create details.BitRate — i.e. quality identity is the bit rate.

Root cause

GlobalConnectSmil.fromCurrentState picked the files for the smil by matching current-state files against desired-state files on FileRef alone:

|> Map.filter (fun fileRef _ -> desiredState.Content |> ContentSet.getFiles |> Map.keys |> Seq.contains fileRef)

FileRef is (PartId option, QualityId), and QualityId comes from the bit rate. The reference is therefore not unique across transcodings — a retranscoding that reuses a bit rate yields the same FileRef.

This is harmless while every desired file is actually uploaded, because the new upload overwrites the current-state entry under that reference. But isContentSupportedByOrigin filters AV1 files out of the upload actions when enableAV1Upload is off:

| MediaPropertiesV2 p -> p.Video.Codec <> "av1" || ctx.EnableAV1Upload()

So the AV1 file at bit rate 1 is never uploaded, and the current-state entry for FileRef(bitrate = 1) still holds the file from the previous transcoding. That stale entry satisfied all three smil filters — right geo restriction, FileRef present in desired content, remote state Completed — and was written into the new smil.

Confirmed empirically. Pre-fix failure:

Expected: ···"deo src="idxf52097762cc_170001_2.mp4" language="no"···
Actual:   ···"deo src="idxf52097762cc_1.mp4" language="nor" audi"···

The smil contained _1.mp4 from the first transcoding where it should have begun at _170001_2.mp4.

The fix

src/Core/Domain/MediaSetTypes.fs, in GlobalConnectSmil.fromCurrentState. The current state's GlobalConnectFile already carries TranscodingVersion, which makes the stale entry identifiable:

let desiredFiles = desiredState.Content |> ContentSet.getFiles
// Matching on file reference alone is not enough: a retranscoding may reuse the same quality (bit rate) for a file
// that is never uploaded, e.g. because its codec is not supported by the origin. The current state then still holds
// the file from the previous transcoding under that reference, and that stale file must not be listed in the smil.
let isDesiredFile fileRef (file: GlobalConnectFileState) =
    match desiredFiles |> Map.tryFind fileRef with
    | Some desiredFile -> file.File.TranscodingVersion >= ContentFile.transcodingVersion desiredFile
    | None -> false
let mediaFiles =
    currentState.Files
    |> Map.filter (fun _ file -> GlobalConnectAccessRestrictions.parse file.File.RemotePath = desiredState.GeoRestriction)
    |> Map.filter isDesiredFile
    |> Map.filter (fun _ file -> file.RemoteState.State = DistributionState.Completed)
    ...

A file left over from an older transcoding is excluded; files at the current version or newer are kept.

Rejected first attempt

The first version compared filenames (remote path last segment vs. desired FileName, both normalized). It broke five GlobalConnectUploadTests cases, which construct current-state files whose remote path does not correspond to the desired filename — e.g. assets/no/open/abc_ID360.mp4 against a desired FileName "abc_ID360.mp4" in a different path shape, plus the geo-restriction move tests.

The version comparison is both more direct and consistent with isFileVersionNewer (GlobalConnectActions.fs:31), which already uses TranscodingVersion to answer the same "is this entry stale" question.

Verification

Check	Result
dotnet build src/Core/Core.fsproj	clean, 0 warnings
Full unit suite (UnitTests.exe)	exit 0, no failures
Target isolated test, Archive + DropFolder	Total: 2, Failed: 0
Same test with fix stashed, container rebuilt	Total: 2, Failed: 2 — the stale-filename diff above

The unit suite has one pre-existing flaky Akka timing failure (GlobalConnect PublishSubtitles job for unchanged subtitles generates events), verified against a stashed baseline as unrelated to this change.

The oddjob container was rebuilt and restarted for each isolated-test run, so the service under test actually ran the code being measured:

cd localservices
docker compose -f docker-compose-isolated-tests.yaml build oddjob
docker compose -f docker-compose-isolated-tests.yaml up -d --no-deps oddjob   # --no-deps: init-mssql-db fails on an already-initialised DB
cd ../tests/IsolatedTests/bin/Debug/net10.0
./IsolatedTests.exe -filter "/*/*/*/*retranscoding*"

Note: dotnet test reports "Zero tests ran" for both test projects (xunit.v3 in-process runner); run the built .exe directly instead.

A full 64-test isolated run showed 8 failures, all infrastructure flakiness from running the suite in parallel — ConnectionsQuotaExceeded for namespace sbemulatorns on the Service Bus emulator and consequent missing playback messages. Re-run individually by the author, they all passed.

Side observation (not changed)

getFeatureFlag (src/Core/Config.fs:538) returns Unchecked.defaultof<'a> rather than defaultValue when a key is missing from AppSettings — the TODO added in 83e9ba20d:

| FeatureFlags.AppSettings settings ->
    if settings.ContainsKey keyName then
        Convert.ChangeType(settings[keyName], typedefof<'a>) |> unbox
    else
        Unchecked.defaultof<'a> //TODO: should we use defaultValue here?

For bool that is false, which is why AV1 upload is disabled in the test environment and why this bug reproduces there. It happens to give the behaviour this test needs, but a flag whose intended default is true would silently read as false. Separate issue from this one.