Skip to content

Instantly share code, notes, and snippets.

@bherila

bherila/temps-mariadb-postgres-handoff.md

Last active June 24, 2026 08:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bherila/438b06bb25d3d8106cab9fffb6475d06 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bherila/438b06bb25d3d8106cab9fffb6475d06 to your computer and use it in GitHub Desktop.
Download ZIP
Temps MariaDB PITR and Postgres upgrade PR handoff
Raw
temps-mariadb-postgres-handoff.md

Temps MariaDB PITR / Postgres Upgrade / CI Handoff

Date: 2026-06-24 UTC (updated end of day) Repo local path: /home/bwh/proj/temps Fork: bherila/temps Upstream: gotempsh/temps

Goal

Unblock upstream PR #138 (MariaDB) by extracting unrelated fixes into separate, focused PRs that can merge independently, then rebase #138 onto main and drop the duplicated commits.

Two focused PRs are now extracted out of #138:

  1. #151 — Postgres major-upgrade hardening.
  2. #152 — CI test isolation (fixes the docker-providers flake/hang that was blocking #138's run).

Current PR Status (all OPEN, all MERGEABLE as of this update)

  • MariaDB PR #138 — gotempsh/temps#138

    • Title: "Add opt-in MariaDB-compatible services"
    • Head: bherila:feature/mariadb-services-upstream-pr → Base: gotempsh:main
    • Blocked on: nothing hard, but should be rebased AFTER #151 and #152 merge so the duplicated Postgres-hardening and any CI changes drop out cleanly.
    • Its last fork run (28071885683) failed only in docker-providers — that failure is what #152 fixes (see below). Not a MariaDB bug.

  • Postgres PR #151 — gotempsh/temps#151

    • Title: "fix(providers): harden postgres major upgrades"
    • Head: bherila:fix/postgres-upgrade-handoff → Base: gotempsh:main
    • Commit: 7a5652a1 fix(providers): harden postgres major upgrades
    • Green fork run when opened: 28072935670.
    • Scope: CHANGELOG.md, crates/temps-providers/src/externalsvc/postgres_upgrade.rs, crates/temps-providers/src/postgres_lifecycle.rs.
    • Status: awaiting upstream review/merge. No action needed from us.

  • CI / test-isolation PR #152 — gotempsh/temps#152 ← NEW

    • Title: "fix(ci): serialize heavy provider backup tests + dedupe port selection"
    • Head: bherila:fix/docker-providers-test-isolation → Base: gotempsh:main
    • Branched from upstream/main (independent of #151/#138; benefits both).
    • Fork CI fully GREEN: https://github.com/bherila/temps/actions/runs/28077627538 (all 4 integration groups + check/fmt/clippy/build/unit). Result + run link posted as PR comments.
    • Status: awaiting upstream review/merge. No action needed from us.

Why PR #152 Exists (the MongoDB "failure")

The docker-providers integration job in #138's run failed with the MongoDB backup test hanging 300s at "Step 5: Backing up to S3". Investigation showed this is NOT a MongoDB product bug and NOT analogous to #151's pg-readiness work.

Root cause: docker-providers ran all four test_{mongodb,postgres,redis,s3}_backup_and_restore_to_s3 tests in PARALLEL. Each spins up a MinIO container + a service container streaming a real wal-g/mongodump backup. Under #138's heavier load they saturated Docker and raced for host ports — the log shows Bind for 0.0.0.0:5433 failed: port is already allocated plus all three backup tests blowing the 300s budget together. The same tests pass on #151 (fewer parallel docker tests). MongoDB was just the most visible victim of contention.

#152 fixes it two ways:

  1. New serial docker-backups CI matrix group runs *_backup_and_restore_to_s3 with --test-threads=1 (mirrors the existing postgres-upgrades group), and --skip backup_and_restore_to_s3 is added to the parallel docker-providers group. File: .github/workflows/rust-tests.yml.
  2. Consolidated the 5 duplicated per-service find_available_port helpers (mongodb/postgres/redis/s3/rustfs + parameter_strategies) into a new crates/temps-providers/src/externalsvc/port_util.rs. Shared sync finder advances a process-wide offset so concurrent allocations diverge (fixes the 5433 collision); a Docker-aware async variant (lifted from rustfs) also skips ports already published by running containers.

No MongoDB product code changed.

What Has Already Been Done

  1. #151 (postgres hardening) opened, green, linked from #138.
  2. #152 (CI test isolation) created from upstream/main, pushed, opened, and validated GREEN on fork CI (run 28077627538). Run link + green result posted as comments on #152.
  3. Local verification for #152: cargo fmt -p temps-providers -- --check, cargo check --lib -p temps-providers (default + --features docker-tests), and the new port_util unit tests all pass.

Immediate Next Checks (tomorrow)

  1. Check whether #151 and #152 have been reviewed/merged upstream: 
    gh pr view 151 --repo gotempsh/temps --json state,mergedAt,reviewDecision
gh pr view 152 --repo gotempsh/temps --json state,mergedAt,reviewDecision
  2. If a fresh fork CI run is wanted on any branch, note the fork's Rust Tests workflow triggers via workflow_dispatch ONLY (not on push/PR): 
    gh workflow run "Rust Tests" --repo bherila/temps --ref <branch>

After #151 AND #152 Merge — Rebase #138

cd /home/bwh/proj/temps
git fetch upstream main
git fetch origin feature/mariadb-services-upstream-pr
git switch feature/mariadb-services-upstream-pr
git rebase upstream/main
# Resolve conflicts by keeping MariaDB/PITR changes and DROPPING:
#   - the duplicate Postgres hardening now in main (from #151)
#   - any docker-providers / port_util / rust-tests.yml changes now in main (from #152)
# Then run focused checks and push --force-with-lease.
git push --force-with-lease origin feature/mariadb-services-upstream-pr

IMPORTANT for the rebase: #152 moved the heavy S3 backup tests into a serial docker-backups CI group selected by the substring backup_and_restore_to_s3. Confirm #138's MariaDB backup test is named test_mariadb_backup_and_restore_to_s3 (or similar matching that substring) so it lands in the serial group automatically. If it is NOT named to match, either rename it or add its name to the docker-backups group filter in .github/workflows/rust-tests.yml. Otherwise it will run in the parallel docker-providers group and can reintroduce the contention/hang #152 fixed.

Expected remaining #138 scope after rebase:

  • MariaDB external service engine/support;
  • MariaDB logical/physical backup engines;
  • S3 binlog archiver and PITR restore/replay;
  • MariaDB health/binlog checks;
  • auto-provisioned default daily base schedule;
  • MariaDB PITR E2E test and CI coverage.

Notes / Decisions

  • The MariaDB PITR E2E uses MinIO in CI as the S3-compatible target, not Garage.
  • Do not add a separate MariaDB-only workflow yet (user: "We don't need #5 yet").
  • Fork Rust Tests workflow is workflow_dispatch-only; dispatch manually.
  • Docker tests require sudo locally for user bwh; GitHub Actions (public repo, unlimited minutes) is the preferred Docker validation path. Run major/docker builds there, not locally.
  • The #138 rebase is intentionally deferred to a separate session.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment