Skip to content

Instantly share code, notes, and snippets.

@tstuefe

tstuefe/coh-by-default-dacapo-runs.md

Last active May 8, 2026 17:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save tstuefe/f27d677052b90ac3eebf458191015c2b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save tstuefe/f27d677052b90ac3eebf458191015c2b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
coh-by-default-dacapo-runs.md

Methodology

I executed all tests with the largest possible datasize for the machine and the benchmark.

I always chose 4 warmup iterations. That was deemed sufficient in [1] but in hindsight it may have been too small.

The JVM is a release build from Coleens' COH-by-default PR.

The max. heap size was chosen to be 2*lifeset size. The lifeset size I determined experimentally beforehand.

All other options default (also no -Xms, no pretouching etc) apart from +/- COH.

[1] Blackburn et al. "Rethinking Java Performance Analysis", ASPLOS 2025

tl;dr Results/Regressions

Many benchmarks showed too high variations for my liking, and I may have to repeat these tests, possibly with longer warmup times. I also may have to repeat them with a large heap size to remove GC from the equation - the current results are running with a quite low ceiling, which was deliberate, but it may have increased "wobblyness".

The only clear negative results are on my machines:

  • sunflow (between 3 and 5% WCT increase)
  • jython (between 1 and 3% WCT increase)

Possibly negative but still too w obbly:

  • grapchi (one machineA I see a 3-4% WCT increase, on the other nothing)

Clear wins:

  • biojava (3-5% WCT, ~5% avg heap size reduction)
  • h2 (unsurprisingly) (3-5% wct, 5-7% heap size reduction)
  • spring shows reduction in WCT and latency, but only on machineA
  • xalan (3-4% WCT)

Machines

  • "machineA" => AMD Ryzen 9 5950X 16-Core Processor, 64GB main memory, Fedora 43
  • "machineB" => Intel(R) Core(TM) i7-4770, 8GB main memory, Fedora Server 44

Metrics

  • WCT: wallclock time
  • P99s: Latency (simple)
  • P99m: Latency (metered over 100ms window)
  • RSS: time-weighted area-under-curve of the full RSS usage measured at periodic intervals over the full length of the benchmark (including warmup)
  • HeapSz: time-weighted area-under-curve of the committed heap size, deduced from gc logs
  • GCs: number of GCs

Avrora

(Note: high deviation between runs of about 2%)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-avrora-large-openjdk-27-jep-534-xmx38m-nocoh-20260507-112645 62.6 - - 0.126 510 21.1
machineA-dacapo-avrora-large-openjdk-27-jep-534-xmx38m-coh-20260507-150844 62.7 - - 0.183 497 20.0
Delta (percent) +0.16 - - +45.24 -2.55 -5.21
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-avrora-large-openjdk-27-jep-534-xmx38m-nocoh-20260507-124723 43.2 - - - 578 20.2
machineB-dacapo-avrora-large-openjdk-27-jep-534-xmx38m-coh-20260507-155729 43.1 - - - 467 19.5
Delta (percent) -0.23 - - - -19.20 -3.47

Batik

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-batik-large-openjdk-27-jep-534-xmx5200m-nocoh-20260508-144133 83.1 - - 5.417 258 3274.7
machineA-dacapo-batik-large-openjdk-27-jep-534-xmx5200m-coh-20260508-143355 83.3 - - 5.370 236 3160.0
Delta (percent) +0.24 - - -0.87 -8.53 -3.50

Biojava

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-biojava-large-openjdk-27-jep-534-xmx3592m-nocoh-20260507-113200 117.6 - - 2.963 638 1816.0
machineA-dacapo-biojava-large-openjdk-27-jep-534-xmx3592m-coh-20260507-151400 111.9 - - 2.741 619 1709.3
Delta (percent) -4.85 - - -7.49 -2.98 -5.88
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-biojava-large-openjdk-27-jep-534-xmx3592m-nocoh-20260507-125101 133.8 - - - 534 1875.8
machineB-dacapo-biojava-large-openjdk-27-jep-534-xmx3592m-coh-20260507-160106 129.1 - - - 488 1844.2
Delta (percent) -3.51 - - - -8.61 -1.68

Eclipse

(Note: high deviation between runs of about 1.5%)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-eclipse-large-openjdk-27-jep-534-xmx274m-nocoh-20260507-205152 27.6 - - 0.770 1176 150.0
machineA-dacapo-eclipse-large-openjdk-27-jep-534-xmx274m-coh-20260508-001126 27.6 - - 0.762 1093 149.1
Delta (percent) +0.00 - - -1.04 -7.06 -0.60
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-eclipse-large-openjdk-27-jep-534-xmx274m-nocoh-20260507-130238 31.5 - - - 1123 155.3
machineB-dacapo-eclipse-large-openjdk-27-jep-534-xmx274m-coh-20260507-161146 31.1 - - - 1033 150.3
Delta (percent) -1.27 - - - -8.01 -3.22

fop

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-fop-default-openjdk-27-jep-534-xmx60m-nocoh-20260508-144842 0.6 - - - 103 23.6
machineA-dacapo-fop-default-openjdk-27-jep-534-xmx60m-coh-20260508-144112 0.6 - - - 92 22.9
Delta (percent) +0.00 - - - -10.68 -2.97

graphchi

(Note: very high variations between runs even on the most stable machines)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-graphchi-large-openjdk-27-jep-534-xmx2042m-nocoh-20260507-114446 150.7 - - 2.113 1217 1415.1
machineA-dacapo-graphchi-large-openjdk-27-jep-534-xmx2042m-coh-20260507-152619 156.0 - - 2.118 1111 1424.0
Delta (percent) +3.52 - - +0.24 -8.71 +0.63
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-graphchi-large-openjdk-27-jep-534-xmx2042m-nocoh-20260507-205642 238.7 - - - 1279 1400.6
machineB-dacapo-graphchi-large-openjdk-27-jep-534-xmx2042m-coh-20260507-235921 239.1 - - - 1218 1396.2
Delta (percent) +0.17 - - - -4.77 -0.31

h2

These tests are very heap intensive. Consequently, we see big improvements stemming from smaller object size and denser

large

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-h2-large-openjdk-27-jep-534-xmx20000m-nocoh-20260507-115740 116.9 8.1 10.3 19.158 473 11361.8
machineA-dacapo-h2-large-openjdk-27-jep-534-xmx20000m-coh-20260507-153920 111.7 7.7 9.5 18.995 434 10458.9
Delta (percent) -4.45 -4.94 -7.77 -0.85 -8.25 -7.95

very large

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-h2-vlarge-openjdk-27-jep-534-xmx48608m-nocoh-20260507-121928 359.8 11.0 12.7 47.082 929 26430.3
machineA-dacapo-h2-vlarge-openjdk-27-jep-534-xmx48608m-coh-20260507-161131 350.9 10.6 12.1 46.570 872 24960.3
Delta (percent) -2.47 -3.64 -4.72 -1.09 -6.14 -5.56

h2o

(Note: high deviation between runs of about 2%)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-h2o-large-openjdk-27-jep-534-xmx3710m-nocoh-20260507-143229 131.8 - - 4.073 3889 2279.3
machineA-dacapo-h2o-large-openjdk-27-jep-534-xmx3710m-coh-20260507-160032 127.9 - - 4.098 3866 2259.9
Delta (percent) -2.96 - - +0.61 -0.59 -0.85
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-h2o-large-openjdk-27-jep-534-xmx3710m-nocoh-20260507-153153 146.1 - - - 1189 1987.9
machineB-dacapo-h2o-large-openjdk-27-jep-534-xmx3710m-coh-20260507-163611 134.8 - - - 1142 1940.4
Delta (percent) -7.73 - - - -3.95 -2.39

jme

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-jme-large-openjdk-27-jep-534-xmx128m-nocoh-20260507-144320 133.5 17.2 17.2 0.273 29 15.6
machineA-dacapo-jme-large-openjdk-27-jep-534-xmx128m-coh-20260507-170918 133.5 17.2 17.2 0.261 37 10.6
Delta (percent) +0.00 +0.00 +0.00 -4.40 +27.59 -32.05
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-jme-large-openjdk-27-jep-534-xmx128m-nocoh-20260507-154402 135.7 17.5 17.5 - 36 11.5
machineB-dacapo-jme-large-openjdk-27-jep-534-xmx128m-coh-20260507-164825 135.7 17.5 17.5 - 36 10.5
Delta (percent) +0.00 +0.00 +0.00 - +0.00 -8.70

jython

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-jython-large-openjdk-27-jep-534-xmx242m-nocoh-20260507-131822 95.7 - - 0.672 5354 79.6
machineA-dacapo-jython-large-openjdk-27-jep-534-xmx242m-coh-20260507-172028 97.1 - - 0.645 5111 74.7
Delta (percent) +1.46 - - -4.02 -4.54 -6.16
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-jython-large-openjdk-27-jep-534-xmx242m-nocoh-20260507-132608 113.0 - - - 6129 72.2
machineB-dacapo-jython-large-openjdk-27-jep-534-xmx242m-coh-20260507-165946 115.4 - - - 6093 66.9
Delta (percent) +2.12 - - - -0.59 -7.34

luindex

(Note: high deviation between runs of about 2%)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-luindex-large-openjdk-27-jep-534-xmx46m-nocoh-20260507-225855 392.3 - - 0.474 92649 30.3
machineA-dacapo-luindex-large-openjdk-27-jep-534-xmx46m-coh-20260508-021719 388.6 - - 0.477 89990 30.3
Delta (percent) -0.94 - - +0.63 -2.87 +0.00
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-luindex-large-openjdk-27-jep-534-xmx46m-nocoh-20260507-133602 473.9 - - - 91945 30.2
machineB-dacapo-luindex-large-openjdk-27-jep-534-xmx46m-coh-20260507-170952 465.9 - - - 89485 30.3
Delta (percent) -1.69 - - - -2.68 +0.33

lusearch

(Note: machineA: extremely high deviation between runs, need to repeat this)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-lusearch-large-openjdk-27-jep-534-xmx136m-nocoh-20260507-141607 395.1 2.3 7.3 - 56302 93.1
machineB-dacapo-lusearch-large-openjdk-27-jep-534-xmx136m-coh-20260507-174933 408.4 2.3 7.3 - 55000 91.4
Delta (percent) +3.37 +0.00 +0.00 - -2.31 -1.83

pmd

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-pmd-large-openjdk-27-jep-534-xmx10102m-nocoh-20260507-140752 12.4 - - 9.421 74 4536.3
machineA-dacapo-pmd-large-openjdk-27-jep-534-xmx10102m-coh-20260507-181204 12.3 - - 9.374 72 4253.0
Delta (percent) -0.81 - - -0.50 -2.70 -6.25

spring

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-spring-large-openjdk-27-jep-534-xmx150m-nocoh-20260507-140858 38.2 44.1 45.4 0.629 14862 101.2
machineA-dacapo-spring-large-openjdk-27-jep-534-xmx150m-coh-20260507-181309 36.4 42.1 43.3 0.647 13547 98.0
Delta (percent) -4.71 -4.54 -4.63 +2.86 -8.85 -3.16
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-spring-large-openjdk-27-jep-534-xmx150m-nocoh-20260507-230407 88.8 19.1 21.8 - 14639 98.7
machineB-dacapo-spring-large-openjdk-27-jep-534-xmx150m-coh-20260508-020753 88.0 19.1 22.3 - 13234 96.1
Delta (percent) -0.90 +0.00 +2.29 - -9.60 -2.63

sunflow

(Note: extremely high deviation between runs, need to repeat this)

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-sunflow-large-openjdk-27-jep-534-xmx294m-nocoh-20260507-141218 61.7 - - 0.449 10484 208.2
machineA-dacapo-sunflow-large-openjdk-27-jep-534-xmx294m-coh-20260507-181625 65.0 - - 0.444 8876 206.9
Delta (percent) +5.35 - - -1.11 -15.34 -0.62
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-sunflow-large-openjdk-27-jep-534-xmx294m-nocoh-20260507-145650 127.0 - - - 10670 196.6
machineB-dacapo-sunflow-large-openjdk-27-jep-534-xmx294m-coh-20260507-183118 132.4 - - - 9300 195.3
Delta (percent) +4.25 - - - -12.84 -0.66

tomcat

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-tomcat-large-openjdk-27-jep-534-xmx60m-nocoh-20260507-141641 47.3 9.2 9.3 0.449 14017 41.7
machineA-dacapo-tomcat-large-openjdk-27-jep-534-xmx60m-coh-20260507-182049 47.1 9.2 9.3 0.473 13347 41.1
Delta (percent) -0.42 +0.00 +0.00 +5.35 -4.78 -1.44
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-tomcat-large-openjdk-27-jep-534-xmx60m-nocoh-20260507-150744 186.7 9.3 9.4 - 13618 39.0
machineB-dacapo-tomcat-large-openjdk-27-jep-534-xmx60m-coh-20260507-184223 187.0 9.3 9.4 - 13019 38.5
Delta (percent) +0.16 +0.00 +0.00 - -4.40 -1.28

xalan

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-xalan-large-openjdk-27-jep-534-xmx30m-nocoh-20260507-142106 41.5 - - 0.228 82797 22.7
machineA-dacapo-xalan-large-openjdk-27-jep-534-xmx30m-coh-20260507-182513 39.8 - - 0.239 78602 22.3
Delta (percent) -4.10 - - +4.82 -5.07 -1.76
name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineB-dacapo-xalan-large-openjdk-27-jep-534-xmx30m-nocoh-20260507-233844 31.7 - - - 23213 14.0
machineB-dacapo-xalan-large-openjdk-27-jep-534-xmx30m-coh-20260507-185826 30.7 - - - 21671 13.8
Delta (percent) -3.15 - - - -6.64 -1.43

zxing

name WCT(s) P99s(ms) P99m(ms) RSS(GB) GCs HeapSz(MB)
machineA-dacapo-zxing-default-openjdk-27-jep-534-xmx340m-nocoh-20260508-144853 1.1 - - - 104 95.9
machineA-dacapo-zxing-default-openjdk-27-jep-534-xmx340m-coh-20260508-144123 1.1 - - - 111 92.8
Delta (percent) +0.00 - - - +6.73 -3.23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment