pgjdbc PR #4143 (Simplify build) — gradle-profiler benchmark: scenarios, scripts, raw CSV results

README.md

Benchmark: pgjdbc PR #4143 ("Simplify build")

Check the speed-up the PR claims (build-logic tasks 92 → 18; cold build ~31s → ~15s) and confirm that everyday development scenarios do not regress.

What is compared

• baseline = 301cfb7a5 (the PR's merge-base with master, i.e. master without the PR) → /tmp/pgjdbc-bench/baseline
• pr-4143 = 3cfca4f4b → /tmp/pgjdbc-bench/pr

baseline is the merge-base rather than the master tip: the only commits between them are docs/test changes that do not touch build-logic, so this isolates the PR's effect alone.

Why two sets of tools

The PR's saving lives in compiling the build-logic included build. gradle-profiler cannot delete arbitrary directories between iterations, so it cannot on its own reproduce a fully cold build-logic compile (where the 31→15 figure comes from). The measurement therefore splits in two:

1. gradle-profiler — statistically sound scenarios (warm-ups + N iterations, mean/stddev/CI):

   • config_help — configuration only, on a cold daemon (JVM start + loading the build-logic plugin classpath + configuration). This is where the PR's wider classpath shows up.
   • compile_cold — :postgresql:compileJava, cold daemon, warm outputs. This is where the "92 → 18 tasks" claim shows up.
   • clean_compile — clean + compileJava on a warm daemon. Control: the PR must not slow down compiling the main code.
   • edit_buildlogic — edit a build-logic Kotlin file between runs. The closest the profiler can get to "build-logic changed". Note: the file path differs between branches (baseline uses the java-comment-preprocessor module, the PR uses the single conventions module), so the scenario lives in mutate-baseline.scenarios / mutate-pr.scenarios. This is also where the god-module downside surfaces: in the PR an edit to one file recompiles the whole conventions module, against only a small module on baseline.

2. cold-buildlogic.sh — a coarse but direct measure of a cold build-logic compile: it stops the daemon, deletes the build-logic outputs, and times :postgresql:compileJava. This reproduces the claimed 31→15.

Running

# 1) gradle-profiler (≈15–30 min total; the first run downloads Gradle 9.5.1 and the plugins)
bash /tmp/pgjdbc-bench/run.sh

# 2) Cold build-logic (≈3 runs per branch)
bash /tmp/pgjdbc-bench/cold-buildlogic.sh 3

gradle-profiler reports: out-baseline/benchmark.html and out-pr/benchmark.html (plus benchmark.csv for collating into one table).

How to read the results

• Compare same-named scenarios between out-baseline and out-pr.
• Look at the standard deviation and confidence interval, not just the mean: the claimed "31s→15s" comes from a single run with no spread, whereas the spread is visible here.
• To answer "why" (confirm the hypothesis about redundant Kotlin compilations), add --profile jfr or --profile async-profiler to the run and read the flame graph.

baseline-benchmark.csv

baseline.scenarios

// pgjdbc PR #4143 benchmark — version-agnostic scenarios (no source mutation).
// Safe to run unchanged on both the baseline and the PR worktree.
//
// What each scenario isolates:
//   config_help   -> pure configuration: settings + included-build (build-logic) plugin
//                    loading + project configuration, with a COLD daemon each run so JVM
//                    start and build-logic classpath loading are counted. No compilation.
//   compile_cold  -> :postgresql:compileJava on a cold daemon, build outputs already warm.
//                    Shows startup + configuration + up-to-date checking for the real graph
//                    (this is where the "92 -> 18 build-logic tasks" claim shows up).
//   clean_compile -> clean + compileJava on a WARM daemon. build-logic stays compiled;
//                    measures main-source compile cost. Control: PR must not regress it.

config_help {
    title = "Configuration only (cold daemon)"
    tasks = ["help"]
    daemon = cold
    warm-ups = 2
    iterations = 10
    gradle-args = ["--quiet"]
}

compile_cold {
    title = "compileJava (cold daemon, warm outputs)"
    tasks = [":postgresql:compileJava"]
    daemon = cold
    warm-ups = 2
    iterations = 6
    gradle-args = ["--quiet"]
}

clean_compile {
    title = "clean + compileJava (warm daemon)"
    cleanup-tasks = ["clean"]
    tasks = [":postgresql:compileJava"]
    daemon = warm
    warm-ups = 2
    iterations = 5
    gradle-args = ["--quiet"]
}
// BASELINE-only: edit a build-logic Kotlin source between runs, then compileJava.
// This is the closest gradle-profiler can get to "build-logic changed" cost:
// the mutated module's Kotlin recompiles and the whole build reconfigures.
//
// On the baseline layout the file lives in the small java-comment-preprocessor
// module, so only that module recompiles.

edit_buildlogic {
    title = "Edit build-logic Kotlin, then compileJava (baseline layout)"
    tasks = [":postgresql:compileJava"]
    apply-abi-change-to = ["build-logic/java-comment-preprocessor/src/main/kotlin/buildlogic/JavaCommentPreprocessorTask.kt"]
    warm-ups = 3
    iterations = 8
    gradle-args = ["--quiet"]
}

cold-buildlogic.sh

#!/usr/bin/env bash
# Reproduce the headline "cold build" number (~31s -> ~15s) that gradle-profiler
# cannot model on its own: a from-scratch build-logic compilation.
#
# For each version we stop the daemon and delete the included build's outputs, so
# Gradle must recompile every build-logic Kotlin module before it can configure the
# main build. Then we time `:postgresql:compileJava`.
#
# Usage: bash cold-buildlogic.sh [runs]   (default 3 runs per version)

set -u
RUNS="${1:-3}"
BASE=/tmp/pgjdbc-bench

now() { python3 -c 'import time; print(time.time())'; }

bench() {
  local name="$1" dir="$2"
  echo "## $name ($dir)"
  for i in $(seq 1 "$RUNS"); do
    ( cd "$dir" && ./gradlew --stop >/dev/null 2>&1 )
    # Wipe included-build outputs so build-logic Kotlin must recompile from scratch.
    rm -rf "$dir"/build-logic/*/build "$dir"/build-logic/.gradle "$dir"/build-logic/*/.gradle \
           "$dir"/build-logic-commons/*/build "$dir"/build-logic-commons/.gradle 2>/dev/null
    local t0 t1
    t0=$(now)
    ( cd "$dir" && ./gradlew --quiet --no-daemon :postgresql:compileJava >/dev/null 2>&1 )
    local rc=$?
    t1=$(now)
    printf '  run %d: %5.1fs (exit %d)\n' "$i" "$(python3 -c "print($t1-$t0)")" "$rc"
  done
}

bench "baseline (master, no PR)" "$BASE/baseline"
bench "pr-4143"                   "$BASE/pr"

cold-nocache.sh

#!/usr/bin/env bash
# True from-scratch build-logic compile: build cache OFF so Kotlin really recompiles.
set -u
RUNS="${1:-3}"; BASE=/tmp/pgjdbc-bench
now(){ python3 -c 'import time;print(time.time())'; }
bench(){ local name="$1" dir="$2"; echo "## $name"
  for i in $(seq 1 "$RUNS"); do
    ( cd "$dir" && ./gradlew --stop >/dev/null 2>&1 )
    rm -rf "$dir"/build-logic/*/build "$dir"/build-logic/.gradle "$dir"/build-logic/*/.gradle \
           "$dir"/build-logic-commons/*/build "$dir"/build-logic-commons/.gradle 2>/dev/null
    local t0 t1; t0=$(now)
    ( cd "$dir" && ./gradlew --quiet --no-daemon --no-build-cache :postgresql:compileJava >/dev/null 2>&1 ); local rc=$?
    t1=$(now); printf '  run %d: %5.1fs (exit %d)\n' "$i" "$(python3 -c "print($t1-$t0)")" "$rc"
  done; }
bench "baseline (master, no PR)" "$BASE/baseline"
bench "pr-4143"                  "$BASE/pr"

common.scenarios

// pgjdbc PR #4143 benchmark — version-agnostic scenarios (no source mutation).
// Safe to run unchanged on both the baseline and the PR worktree.
//
// What each scenario isolates:
//   config_help   -> pure configuration: settings + included-build (build-logic) plugin
//                    loading + project configuration, with a COLD daemon each run so JVM
//                    start and build-logic classpath loading are counted. No compilation.
//   compile_cold  -> :postgresql:compileJava on a cold daemon, build outputs already warm.
//                    Shows startup + configuration + up-to-date checking for the real graph
//                    (this is where the "92 -> 18 build-logic tasks" claim shows up).
//   clean_compile -> clean + compileJava on a WARM daemon. build-logic stays compiled;
//                    measures main-source compile cost. Control: PR must not regress it.

config_help {
    title = "Configuration only (cold daemon)"
    tasks = ["help"]
    daemon = cold
    warm-ups = 2
    iterations = 10
    gradle-args = ["--quiet"]
}

compile_cold {
    title = "compileJava (cold daemon, warm outputs)"
    tasks = [":postgresql:compileJava"]
    daemon = cold
    warm-ups = 2
    iterations = 6
    gradle-args = ["--quiet"]
}

clean_compile {
    title = "clean + compileJava (warm daemon)"
    cleanup-tasks = ["clean"]
    tasks = [":postgresql:compileJava"]
    daemon = warm
    warm-ups = 2
    iterations = 5
    gradle-args = ["--quiet"]
}

pr-benchmark.csv

pr.scenarios

// pgjdbc PR #4143 benchmark — version-agnostic scenarios (no source mutation).
// Safe to run unchanged on both the baseline and the PR worktree.
//
// What each scenario isolates:
//   config_help   -> pure configuration: settings + included-build (build-logic) plugin
//                    loading + project configuration, with a COLD daemon each run so JVM
//                    start and build-logic classpath loading are counted. No compilation.
//   compile_cold  -> :postgresql:compileJava on a cold daemon, build outputs already warm.
//                    Shows startup + configuration + up-to-date checking for the real graph
//                    (this is where the "92 -> 18 build-logic tasks" claim shows up).
//   clean_compile -> clean + compileJava on a WARM daemon. build-logic stays compiled;
//                    measures main-source compile cost. Control: PR must not regress it.

config_help {
    title = "Configuration only (cold daemon)"
    tasks = ["help"]
    daemon = cold
    warm-ups = 2
    iterations = 10
    gradle-args = ["--quiet"]
}

compile_cold {
    title = "compileJava (cold daemon, warm outputs)"
    tasks = [":postgresql:compileJava"]
    daemon = cold
    warm-ups = 2
    iterations = 6
    gradle-args = ["--quiet"]
}

clean_compile {
    title = "clean + compileJava (warm daemon)"
    cleanup-tasks = ["clean"]
    tasks = [":postgresql:compileJava"]
    daemon = warm
    warm-ups = 2
    iterations = 5
    gradle-args = ["--quiet"]
}
// PR-only: same mutation as mutate-baseline.scenarios, but the file now lives in the
// single `conventions` module. An ABI change there recompiles the WHOLE conventions
// module (every convention plugin), which is the trade-off of the god-module layout.
// Compare this number against the baseline edit_buildlogic to see incremental cost.

edit_buildlogic {
    title = "Edit build-logic Kotlin, then compileJava (PR layout)"
    tasks = [":postgresql:compileJava"]
    apply-abi-change-to = ["build-logic/conventions/src/main/kotlin/buildlogic/JavaCommentPreprocessorTask.kt"]
    warm-ups = 3
    iterations = 8
    gradle-args = ["--quiet"]
}

RESULTS.md

Benchmark: what does the consolidation actually save?

I measured this PR with gradle-profiler to see where the configuration time goes. Short version: the speed-up is real but small in everyday use, the headline cold-build number only shows up with the build cache disabled (which is not how this project builds), and editing build-logic itself gets noticeably slower.

Setup

• baseline = 301cfb7a5 (the PR's merge-base, i.e. master without this PR); PR = 3cfca4f4b
• gradle-profiler 0.24.0, Gradle 9.5.1, JDK 21 (Corretto), macOS arm64
• Each cell is the median of the measured runs (after warm-ups); sd is the population standard deviation, n the sample count
• cold daemon = a fresh daemon per run; warm daemon = reused daemon; cold-build-logic rows stop the daemon and delete the build-logic outputs before each run
• Full scenarios, scripts, and raw CSVs: gist

Results

Scenario	master	PR #4143	Δ median
Configuration only (help, cold daemon)	9.82s (sd 1.47, n=10)	8.47s (sd 0.51)	−14%
:postgresql:compileJava (cold daemon, warm outputs)	9.27s (sd 0.41, n=6)	8.39s (sd 0.70)	−9%
clean + compileJava (warm daemon)	0.92s (sd 0.06, n=5)	0.58s (sd 0.03)	−37% (≈0.34s)
Edit a build-logic Kotlin file, then compileJava	3.74s (sd 0.63, n=9)	7.24s (sd 0.77)	+94%
Cold build-logic compile, build cache on	~11.6s	~10.3s	~−1.3s
Cold build-logic compile, build cache off	~36.5s	~26.1s	−28% (≈10s)

Reading the numbers

1. Everyday work shows no meaningful difference. On a warm daemon with build-logic untouched, the gap is about 0.34s. The percentage looks large only because the absolute time is tiny; the daemon already holds the configured model, so applying fewer plugins barely moves the wall clock.

2. Editing build-logic gets slower, but that is rare. A change to a build-logic source goes from ~3.7s to ~7.2s (warm-up runs reached ~9–10s). The cause is exactly the trade-off of one module: in master an edit recompiles the small java-comment-preprocessor module; in the PR it recompiles the whole conventions module. The common case here is a plugin version bump in conventions/build.gradle.kts, which hits the same path. It is infrequent, so the impact in practice is low, but it is a real regression rather than a wash.

3. With the build cache on, the cold build is almost a tie (~11.6s vs ~10.3s). The Kotlin outputs are cached, so the "redundant compilation passes" the PR removes were not being re-run anyway.

4. The 50%-class win needs the build cache turned off (~36.5s → ~26.1s, about 10s). That is a non-default mode: this project pushes to a remote S3 build cache on CI (com.github.burrunan.s3-build-cache, isPush = isCiServer && …), so even a fresh checkout pulls the build-logic classes from the cache. A truly cold, cache-less compile is a corner case, not the path most builds take.

Takeaway

The refactor is a genuine simplification and trims cold configuration by ~10–14%, but the build-time argument is weaker than the description suggests: with the build cache the project already relies on, the saving is well under a second in normal use, and the one workflow that gets measurably worse — editing build-logic — costs roughly 2× more. Worth landing on its own merits (fewer modules, less indirection); the performance framing is the part I'd soften.

One maintenance caveat: inlining shared logic instead of factoring it into a plugin invites duplication. Dropping the build-logic.repositories plugin turned one repository block into two inline copies, in the root build and in build-logic.java, and they already diverged within this PR: the root build lost releasesOnly() and the Sonatype snapshots repository. Expect that drift to recur wherever a former plugin is inlined.

How to reproduce

# baseline (merge-base) and PR head in separate worktrees
git worktree add --detach /tmp/bench/baseline 301cfb7a5
git worktree add --detach /tmp/bench/pr 3cfca4f4b

# gradle-profiler scenarios per layout (build-logic paths differ between branches)
gradle-profiler --benchmark --project-dir /tmp/bench/baseline \
  --scenario-file baseline.scenarios config_help compile_cold clean_compile edit_buildlogic
gradle-profiler --benchmark --project-dir /tmp/bench/pr \
  --scenario-file pr.scenarios config_help compile_cold clean_compile edit_buildlogic

# cold build-logic: stop daemon, wipe build-logic outputs, time compileJava
#   (run once with --build-cache and once with --no-build-cache)

gradle-profiler cannot wipe an included build's outputs between iterations, so the cold-build-logic rows use a small shell wrapper rather than a profiler scenario.

run.sh

#!/usr/bin/env bash
# Orchestrates the gradle-profiler benchmarks for PR #4143.
# Runs the version-agnostic scenarios plus the per-layout build-logic-edit scenario
# on each worktree, into separate output dirs.
#
# Usage: bash run.sh
set -eu
BASE=/tmp/pgjdbc-bench
GP=$(command -v gradle-profiler)

echo ">>> baseline: common + build-logic edit"
"$GP" --benchmark \
  --project-dir "$BASE/baseline" \
  --scenario-file "$BASE/baseline.scenarios" \
  --output-dir "$BASE/out-baseline" \
  config_help compile_cold clean_compile edit_buildlogic

echo ">>> pr-4143: common + build-logic edit"
"$GP" --benchmark \
  --project-dir "$BASE/pr" \
  --scenario-file "$BASE/pr.scenarios" \
  --output-dir "$BASE/out-pr" \
  config_help compile_cold clean_compile edit_buildlogic

echo
echo "Reports:"
echo "  baseline: $BASE/out-baseline/benchmark.html  (+ benchmark.csv)"
echo "  pr-4143 : $BASE/out-pr/benchmark.html        (+ benchmark.csv)"