Scalability Test: Spring vs Quarkus — Local, 20 Apr 2026

Environment

Machine: AMD Ryzen 9 7950X 16-Core (32 CPUs), 64 GB RAM
OS: Fedora Linux 43, kernel 6.19.11-200.fc43.x86_64
Java: 25.0.2-tem
Quarkus: 3.34.3
Spring Boot 4: 4.0.5
CPU pinning: app=0-3, db=4-6, load-gen=7-9, monitor=10, otel=11-13
Network: host networking (--use-container-host-network)
DB connection pool: 50 (default)
Warmup: 2 min wrk, 30s sleep, then 30s measurement
Iterations: 1 per connection count (except 1-conn Q3-JVM/Q3-Virtual: 3 iterations)

Runtimes

Runtime	Description
quarkus3-jvm	Quarkus 3.34.3, executor threads (Vert.x + JBoss Threads)
quarkus3-virtual	Quarkus 3.34.3, virtual threads (Loom)
spring4-jvm	Spring Boot 4.0.5, Tomcat thread pool
spring4-virtual	Spring Boot 4.0.5, virtual threads

TPS Scaling Curve

Conn	Q3-JVM	Q3-Virtual	S4-JVM	S4-Virtual
1	3,132	2,744	2,206	2,035
2	6,014	5,060	4,373	3,865
3	8,241	7,069	6,354	5,728
4	9,234	8,165	7,542	6,650
5	10,286	9,798	8,219	7,839
10	13,517	13,122	9,834	10,738
50	15,765	15,298	10,536	11,826
100	15,690	15,041	9,901	11,103

Latency (avg)

Conn	Q3-JVM	Q3-Virtual	S4-JVM	S4-Virtual
1	318 μs	363 μs	452 μs	490 μs
2	331 μs	394 μs	456 μs	516 μs
3	363 μs	423 μs	471 μs	522 μs
4	431 μs	488 μs	528 μs	599 μs
5	484 μs	508 μs	606 μs	636 μs
10	737 μs	759 μs	1.01 ms	929 μs
50	3.16 ms	3.26 ms	4.74 ms	4.22 ms
100	6.35 ms	6.63 ms	10.09 ms	8.99 ms

Observations

1. Per-request baseline (1 connection, zero contention)

Q3-JVM is the fastest: 318 μs / 3,132 TPS
Q3-Virtual: +14% overhead (363 μs)
S4-JVM: +42% slower than Q3-JVM (452 μs)
S4-Virtual: +54% slower than Q3-JVM (490 μs)

2. Quarkus scaling: JVM and Virtual converge at saturation

At 1 conn: Q3-JVM is 14% faster than Q3-Virtual
At 50 conn: gap narrows to 3% (15,765 vs 15,298)
At 100 conn: gap is 4% (15,690 vs 15,041)
Both plateau at ~15,000-15,700 TPS — CPU-bound on 4 cores
Virtual threads add per-request overhead but don't change the plateau shape

3. Spring: virtual threads OVERTAKE platform threads at high concurrency

At 1-5 conn: S4-JVM is faster than S4-Virtual (VT overhead dominates)
At 10 conn: crossover — S4-Virtual (10,738) beats S4-JVM (9,834) by 9%
At 50 conn: S4-Virtual (11,826) beats S4-JVM (10,536) by 12%
At 100 conn: S4-Virtual (11,103) beats S4-JVM (9,901) by 12%
S4-JVM actually regresses from 50→100 connections (10,536 → 9,901)

4. Spring4-JVM degrades at high concurrency

S4-JVM peaks at 50 connections (10,536) then drops at 100 (9,901)
S4-Virtual does not degrade (11,826 at 50 conn, 11,103 at 100 conn)

5. Quarkus vs Spring gap widens at saturation

At 1 conn: Q3-JVM is 42% faster than S4-JVM
At 100 conn: Q3-JVM is 59% faster than S4-JVM
At 100 conn: Q3-JVM is 41% faster than S4-Virtual (Spring's best)
The gap widens because Quarkus has lower per-request CPU cost, and at CPU saturation that directly maps to TPS

6. Plateau analysis

Runtime	Plateau TPS	At conn	Scaling factor (vs 1-conn)
Q3-JVM	~15,700	50	5.02x
Q3-Virtual	~15,300	50	5.58x
S4-Virtual	~11,800	50	5.80x
S4-JVM	~10,500	50	4.77x

S4-JVM has the worst scaling factor (4.77x). S4-Virtual scales better (5.80x).

8-Core Test (50 connections, 21 Apr 2026)

CPU pinning: app=0-7 (NUMA 0), db=8-10 (NUMA 1), load-gen=11-13 (NUMA 1), monitor=14, otel=15

TPS: Host Networking vs Pasta

Runtime	Host Net TPS	Pasta TPS	Pasta overhead
Q3-JVM	25,576	16,280	-36.3%
Q3-Virtual	26,367	16,479	-37.5%
S4-JVM	18,789	15,824	-15.8%
S4-Virtual	21,777	14,736	-32.3%

Latency (avg, 50 connections)

Runtime	Host Net	Pasta
Q3-JVM	—	3.07 ms
Q3-Virtual	—	3.03 ms
S4-JVM	—	3.15 ms
S4-Virtual	2.29 ms	3.39 ms

4-core vs 8-core scaling (host networking, 50 connections)

Runtime	4-core TPS	8-core TPS	Scaling factor
Q3-JVM	15,765	25,576	1.62x
Q3-Virtual	15,298	26,367	1.72x
S4-JVM	10,536	18,789	1.78x
S4-Virtual	11,826	21,777	1.84x

Observations

7. Virtual threads overtake JVM threads at 8 cores

Q3-Virtual (26,367) beats Q3-JVM (25,576) by 3.1% — reversed from 4-core where Q3-JVM was 3% ahead
S4-Virtual (21,777) extends lead over S4-JVM (18,789) to 15.9% (was 12.2% at 4 cores)
With more cores, the per-request VT overhead becomes less significant relative to the I/O overlap advantage

8. Sub-2x scaling from 4→8 cores

All runtimes scale 1.62-1.84x instead of 2x
App CPUs 0-7 are NOT saturated: 87.6% busy for S4-Virtual (host net)
DB CPUs 8-10 are at 66.4% busy — becoming a constraint (was ~45% at 4 cores)
The bottleneck is shifting from app CPU toward DB processing capacity

9. Pasta proxy is a hard ceiling

Pasta is single-threaded and saturates its CPU core at 99.7% (measured separately, see below)
Its ceiling is ~39K req/s at 2.28 GHz, ~80K at full boost clock
For this workload (2 JDBC round-trips per request): theoretical max ~20K TPS at 2.28 GHz, ~40K at full clock
Measured Q3-JVM at 16,280 TPS = ~32.5K round-trips/sec through pasta — consistent with the ceiling
At 8 cores with pasta, app CPUs are only 70% busy — pasta limits throughput before app saturates
All 4 runtimes converge to ~15-16.5K under pasta, erasing most framework differences
The overhead is pure queuing (waiting in line for a single-threaded proxy), not contention or extra CPU cost in the app
S4-JVM (15,824) actually beats S4-Virtual (14,736) under pasta — VT scheduling overhead without CPU saturation benefit
Pasta overhead characterization: /home/forked_franz/IdeaProjects/pasta-overhead/doc/findings.md

Scheduling Analysis: Why Loom Helps Spring but Not Quarkus (100 connections, 4 cores)

Thread model per runtime

Runtime	Poller/Event-loop	Worker model	Active workers
Q3-JVM	4 vert.x-eventloop	101 JBoss executor-thread (platform)	~101
Q3-Virtual	4 vert.x-eventloop + 1 MasterPoller	4 FJ carriers + delayScheduler	10 OS threads
S4-JVM	1 http-nio-8080-Poller	103 http-nio-8080-exec (platform)	~103
S4-Virtual	1 http-nio-8080-Poller + 1 MasterPoller	4 FJ carriers + delayScheduler	7 OS threads

Virtual thread OS threads during steady state (from pidstat, local 100-conn run):

Q3-Virtual (10 active OS threads, from pidstat scaling-others-c100, measurement window only):

4 ForkJoinPool-1-* carriers at ~20% CPU each
1 ForkJoinPool-1-* at ~20% CPU (likely delayScheduler — pidstat truncates the name)
4 vert.x-eventloop at ~11.5% CPU each
1 MasterPoller at 2.2% CPU

S4-Virtual (7 active OS threads, from pidstat scaling-others-c100, measurement window only):

5 ForkJoinPool-1-* threads at 8-9% CPU each (4 carriers + delayScheduler)
1 http-nio-8080-Poller at 23.6% CPU
1 MasterPoller at 1.0% CPU

Context switches (all JVM threads, 20s measurement window, 100 connections)

Methodology: 20s sub-window extracted from the 30s wrk measurement period (identified via mpstat CPU 0 busy→idle transitions, trimmed to avoid ramp effects). Context switches summed across ALL JVM threads (unique TIDs in pidstat cswch section, NF==8 lines). %wait averaged across worker threads only (pidstat CPU section, NF==12 lines).

Runtime	TPS	Vol/s	Nvol/s	Vol/req	Nvol/req	Worker %wait	EL/Poller %wait
Q3-JVM	15,690	47,851	1,201	3.05	0.08	34.6%	75.4%
Q3-Virtual	15,041	24,086	21,087	1.60	1.40	19.3%	14.7%
S4-JVM	9,901	23,140	4,055	2.34	0.41	48.9%	54.7%
S4-Virtual	11,103	15,844	14,860	1.43	1.34	7.4%	30.7%

Interpretation

The numbers above are local (kernel 6.19). The perf-lab (kernel 5.14) shows the same directional pattern — see the cross-environment comparison for the data.

Why Loom helps Spring: S4-JVM has 103 platform threads on 4 cores. Loom collapses these to ~7 OS threads (4 carriers + delayScheduler + poller + MasterPoller — from pidstat). Observations: %wait drops from 48.9% to 7.4%, vol/req drops from 2.34 to 1.43. Result: +12% TPS locally (11,103 vs 9,901), +20.5% on perf-lab (9,179 vs 7,616). The Loom benefit is larger on the perf-lab because it eliminates two sources of waste: (1) HikariCP yield-spin contention (absent locally on EEVDF) and (2) CFS scheduling overhead from 103 threads. The Agroal experiment (see below) confirmed this: with Agroal (no yield-spin), the perf-lab Loom benefit drops to +15.9%, and locally it stays at +11.9%.

With Loom (S4-Virtual), involuntary preemptions per request go UP locally (0.41 → 1.34 nvol/req): carriers never block from the kernel's perspective (virtual thread I/O is handled in userspace), so the kernel must forcefully preempt them.

Why Loom hurts Quarkus: Q3-JVM uses 4 event-loop threads for non-blocking HTTP I/O + executor-thread workers for blocking JDBC. Result: -4% TPS locally (15,041 vs 15,690), -7.7% on perf-lab (12,241 vs 13,265).

The context switch data does NOT fully explain this. Locally, total context switches per request are nearly flat: Q3-JVM 3.13/req vs Q3-Virtual 3.00/req. On the perf-lab, total switches increase more: Q3-JVM 3.54/req vs Q3-Virtual 6.42/req (+81%). The cause of this difference between environments is not known.

Since total switches per request are flat locally, the -4% TPS cost points to overhead beyond kernel context switches. syncjfr profiling (local, multiple runs) identifies two candidate costs. Caveat: syncjfr adds overhead that reverses the TPS ranking — Q3-Virtual beats Q3-JVM under profiling (~14.3K vs ~13.6K), opposite of the unprofiled baseline (15,041 vs 15,690). The profiler hurts Q3-JVM more (101 platform threads generate more JFR events than ~5 carriers). The observations below are relative comparisons within the same profiling setup and consistent across two independent runs, but they cannot be taken as proven causes of the -4% unprofiled cost:

	Q3-JVM	Q3-Virtual	Impact
GC count (210s)	542	714 (+32%)	More stop-the-world pauses
Allocation rate	782 MB/s	1,033 MB/s	+32% from VT infrastructure
Total GC pause	1,489 ms	1,667 ms	+178ms over 210s
`writeBlocking` + `wakeup`	252 (0.8%)	1,311 (3.1%)	5.2x more in Virtual
Context switches/req	3.13	3.00	Flat

Response write path overhead: Q3-Virtual shows VertxOutputStream.writeBlocking (1,209 samples, 2.9%) + SelectedSelectionKeySetSelector.wakeup (759 samples, 1.8%) = 4.7% of wall-clock time. These have no equivalent in Q3-JVM. In Q3-JVM, the executor thread writes the response directly through the event loop. In Q3-Virtual, the virtual thread goes through a blocking write path, then wakes the event-loop selector to actually send it — a VT park + selector wakeup roundtrip per response.

32% higher allocation rate → 32% more GCs: Q3-Virtual allocates ~1,033 MB/s vs Q3-JVM's 782 MB/s (estimated from GC count × heap delta). Same live set (~56 MB after GC), same trigger point (~360 MB before GC), but Q3-Virtual fills it faster. Result: 714 GCs vs 542 in 210s, with 178ms more total pause time (1,667ms vs 1,489ms). The extra pause time alone is small (~0.1% of wall time), but more frequent GCs also cause cache pollution and memory barriers.

Allocation analysis used two complementary tools: async-profiler -e alloc (TLAB-based, captures application allocations with stacks, misses outside-TLAB like StackChunk) and syncjfr ObjectAllocationSample via jfrconv --alloc --total (JVM-level sampling, captures all allocation types including StackChunk).

VT-specific allocation: unpooled heap buffers in response write path. In Quarkus AppendBuffer.append() (independent-projects/vertx-utils/src/main/java/io/quarkus/vertx/utils/AppendBuffer.java, line 130-137):

if (isVirtualThread()) {
    tmpBuf = VertxByteBufAllocator.DEFAULT.heapBuffer(chunkCapacity);  // unpooled heap
} else {
    tmpBuf = PooledByteBufAllocator.DEFAULT.directBuffer(chunkCapacity);  // pooled direct
}

Intentional trade-off: Netty's PooledByteBufAllocator uses ThreadLocal caches that leak with VTs. Cost: 854 MB vs 94 MB in async-profiler alloc (9x). Patching to force pooled direct buffers reduced this to 204 MB but TPS was flat (14,792 vs 14,952). Could be addressed by using Netty 4.1's AdaptiveByteBufAllocator (doesn't use ThreadLocal caches) — available since 4.1.x but not used by Vert.x/Quarkus yet.

VT-specific allocation: StackChunk from Continuation.doYield. Invisible to async-profiler (outside TLAB), visible in syncjfr. Each JDBC blocking I/O freezes the VT's continuation stack into a StackChunk. With Hibernate's 30+ frame deep stacks, each chunk is ~5 KB.

Top allocation stacks (syncjfr --alloc --total, with BigDecimal price fix applied to eliminate FloatToDecimal):

Stack	Q3-JVM	Q3-Virtual	VT-specific?
`Continuation.doYield` → StackChunk	0	14,778 MB	Yes
`PGStream.receiveTupleV3` → byte[]	4,750 MB	5,912 MB	No — JDBC result reading
Hibernate `EntityEntryImpl`	3,732 MB	3,999 MB	No — entity tracking
`StoreFruitPrice` instantiate	3,240 MB	3,964 MB	No — entity creation
Hibernate `EntityHolderImpl`	2,912 MB	3,380 MB	No — persistence context
PG `ByteConverter` → BigDecimal	2,387 MB	2,874 MB	No — result parsing
PG `ByteConverter` → BigInteger	2,309 MB	3,116 MB	No — result parsing

All non-StackChunk allocations are the same application work (Hibernate entity management, JDBC result parsing). The per-stack differences between JVM and Virtual are proportional to syncjfr sampling, not VT-specific. StackChunk is the only VT-specific allocation, accounting for ~14.8 GB out of the 49.5 GB total sampled difference (30%).

GC impact (with BigDecimal fix + AppendBuffer patch): Q3-Virtual still has 25% more GCs (679 vs 542) despite matching TLAB allocation. StackChunk drives the extra GC pressure — inherent to Loom and not fixable at application level.

BigDecimal price fix: StoreFruitPriceDTO uses float price but the entity uses BigDecimal and postgres stores NUMERIC. The mapper calls BigDecimal.floatValue() — a lossy narrowing (1.29 becomes 1.2899999618530273). Jackson then serializes via writeNumber(float) → Float.toString → FloatToDecimal.toString → byte[] allocation. Changing the DTO to BigDecimal eliminates this: writeNumber(BigDecimal) uses BigDecimal.toString() directly. byte[] reduced ~10%, TPS +3% under syncjfr. Applies to all modules.

Data: async-profiler alloc at /tmp/q3-alloc-profile/, patched AppendBuffer at /tmp/q3-virtual-patched/, BigDecimal fix syncjfr at /tmp/q3-bigdecimal-fix/

Additional observations from the same syncjfr data:

Event-loop vol cswch explosion with Loom: Event-loop voluntary switches per thread jump massively when moving from Q3-JVM to Q3-Virtual, on both schedulers:

Runtime EEVDF (local) vol/s EEVDF nvol/s CFS (perf-lab) vol/s CFS nvol/s

Q3-JVM EL 784 ~0 27 713

Q3-Virtual EL 15,573 ~0 5,909 3

Jump 20x — 219x —

(All per event-loop thread, from pidstat measurement window.)

On CFS, Q3-JVM event loops have low vol (27/s) but high nvol (713/s) — they're being preempted by waking executor threads. On EEVDF, Q3-JVM event loops have high vol (784/s) but near-zero nvol — EEVDF protects them from preemption, so they park voluntarily. With Loom, vol explodes on both schedulers. The cause is not determined. A related issue is vert.x#6063 (lock contention between carriers and event loops on synchronized(connection) during response writes), profiled with async-profiler -e lock on a similar setup, but it is not confirmed as the cause here.

Runtime	EEVDF (local) vol/s	EEVDF nvol/s	CFS (perf-lab) vol/s	CFS nvol/s
Q3-JVM EL	784	~0	27	713
Q3-Virtual EL	15,573	~0	5,909	3
Jump	20x	—	219x	—

CPU saturation gap (perf-lab only): Q3-Virtual reaches only 90.1% app CPU vs Q3-JVM's 99.8% (from mpstat above). That ~10% idle gap accounts for most of the -7.7% TPS difference on CFS.

Lock contention analysis (async-profiler -e lock --total, Q3-Virtual with Vert.x PR #6081 applied locally — PR removes synchronized in ConnectionBase.writeToChannel, merged into vertx-core 4.x after 4.5.26 release):

Vert.x fix TPS impact (local, no profiler): Q3-JVM 15,282, Q3-Virtual 14,862, Loom effect -2.7% (improved from -4.1% baseline). The fix eliminates queueForWrite lock contention but exposes a deeper bottleneck.

Lock contention with Vert.x fix (10s, Q3-Virtual):

Lock	Wait time	Stack
`EPollSelectorImpl.wakeup` → `interruptLock`	7,523 ms	VTs calling `execute()` → `wakeup()` → `synchronized(interruptLock)`
`EPollSelectorImpl.clearInterrupt`	1,375 ms	Event loop clearing interrupt → `synchronized(interruptLock)`
`Http1xServerConnection.enqueueRequest`	431 ms	Event loop → `synchronized(connection)` (Quarkus VertxOutputStream)
`Http1xServerConnection.onEnd`	301 ms	Event loop → `synchronized(connection)` (Quarkus VertxOutputStream)
Total	9,634 ms

The NIO selector interruptLock dominates (92%). VTs calling writeToChannel → execute() → wakeup() contend on synchronized(interruptLock) inside EPollSelectorImpl.wakeup(). On JDK 25 (JEP 491), VTs unmount from their carrier when blocked on synchronized — carriers are not stalled. The 9,634 ms is VT wall-clock wait time, not carrier blocking time. The throughput impact depends on whether enough ready VTs exist to keep carriers busy (with 100 connections, they should). This contention adds per-request latency but may not directly limit throughput. See netty#16158. The Http1xServerConnection contention (732 ms) is from Quarkus's VertxOutputStream.write() holding synchronized(request.connection()) across the wakeup chain (quarkus#37907).

Native epoll transport eliminates NIO interruptLock contention. Branch epoll-on-main adds netty-transport-native-epoll + quarkus.vertx.prefer-native-transport: true to both quarkus3 and quarkus3-virtual. Netty's EpollEventLoop uses eventfd directly — no synchronized for wakeup.

Combined with the Vert.x fix (local, no profiler, vertx-core 4.5.26 + cherry-picked edcd07c56 only):

Configuration	Q3-JVM	Q3-Virtual	Loom effect
Baseline (NIO, no fixes)	15,690	15,041	-4.1%
Vert.x fix only (NIO)	15,282	14,862	-2.7%
Native epoll only	15,585	14,678	-5.8%
Epoll + Vert.x fix	15,281	15,622	+2.2%

With both fixes, Loom goes from -4.1% to +2.2% — Q3-Virtual now beats Q3-JVM. The two lock contentions (NIO interruptLock + ConnectionBase.synchronized) were the entire Loom penalty for Quarkus on this workload.

No VT carrier pinning detected (-Djdk.tracePinnedThreads=short on JDK 25.0.2-tem — zero pinning events). JDK 25 does not pin on synchronized (JEP 491).

Lock contention with epoll + vertx fix (async-profiler -e lock --total, 10s, Q3-Virtual):

Lock	NIO + vertx fix	Epoll + vertx fix	Change
NIO `interruptLock`	8,898 ms	22 ms	-99.8%
`Http1xServerConnection`	732 ms	819 ms	~same
Total	9,634 ms	855 ms	-91%

Native epoll eliminated NIO selector contention. The remaining 819 ms is Quarkus's VertxOutputStream.synchronized(connection) — the next fix target (quarkus#37907).

Data: epoll-only at /tmp/q3-jvm-epoll/, epoll+vertx at /tmp/q3-epoll-cleanfix/, lock profile at /tmp/q3-epoll-lock-clean/.

GC fixes do not make Loom beneficial for Quarkus. Three patches were tested in combination (locally, no profiler, branch bigdecimal-fix): (1) per-event-loop VT pool (StackChunk reuse), (2) forced pooled direct buffers in AppendBuffer, (3) BigDecimal price DTO fix. Results:

	Baseline (no fixes)	Combined GC fixes	Change
Q3-JVM	15,690	15,378	-2%
Q3-Virtual	15,041	14,868	-1.1%
Loom effect	-4.1%	-3.3%	slightly better

The combined fixes reduced StackChunk allocation 84% (16,310 → 2,606 MB), GC count -10% (695 → 628), and GC pause -15% (1,747 → 1,373 ms). But Loom still hurts: -3.3% vs -4.1%. The GC overhead is a real cost but not the reason Loom doesn't help Quarkus. The Quarkus event-loop + executor architecture already handles concurrency efficiently — Loom adds VT scheduling overhead (mount/unmount, ForkJoinPool dispatch, event-loop reschedule) that cancels any benefit from fewer OS threads.

Data: per-EL pool at /tmp/q3-vt-per-el/, combined at /tmp/q3-vt-combined/, no-profiler at /tmp/q3-vt-combined-noprof/, lock-investigation/ and lock-investigation-virtual/.

Per-request app CPU demand (derived from the utilization law: S_app = cores × U_app / TPS):

Runtime	Local (100-conn)	Perf-lab
Q3-JVM	0.255 ms (4 × 0.998 / 15,690)	0.301 ms (4 × 0.998 / 13,265)
S4-JVM	0.404 ms (4 × 1.0 / 9,901)	0.478 ms (4 × 0.911 / 7,616)
Ratio (S4/Q3)	1.58x	1.59x

Important: these numbers are the sum of app CPU phases only, not the full transaction service time. Each request ping-pongs between app cores and DB cores: app CPU (parse, Hibernate setup) → DB (query 1) → app CPU (process, build query 2) → DB (query 2) → app CPU (hydrate, serialize, respond). The utilization law sums all app CPU phases into a single number, hiding the multi-stage structure. It predicts TPS at saturation (where DB wait is hidden by thread parallelism: TPS ≈ cores / S_app) but cannot be used to reason about latency, queueing, or how stages interact with scheduling. At 1 connection (no queueing), the full transaction time is visible: Q3-JVM local = 318μs, of which DB RTT accounts for ~113μs and the remainder includes app CPU + wrk↔app overhead.

TPS ratios across environments (local 100-conn and perf-lab avg):

Pair	Local	Perf-lab	Stable?
Q3-JVM / S4-JVM	1.58x (15,690/9,901)	1.74x (13,265/7,616)	No — gap widens on perf-lab. Explained: HikariCP contention. With Agroal: local 1.62x, perf-lab 1.63x — stable.
Q3-JVM / S4-Virtual	1.41x (15,690/11,103)	1.44x (13,265/9,179)	Yes
Q3-Virtual / S4-Virtual	1.35x (15,041/11,103)	1.33x (12,241/9,179)	Yes

The JVM/JVM ratio widens on the perf-lab (1.74x vs 1.58x). Once both use Loom (bottom two rows), the ratio stabilizes at ~1.34x. The JVM/JVM ratio difference is caused by HikariCP yield-spin contention on the perf-lab: replacing HikariCP with Agroal gives local 1.62x, perf-lab 1.63x — the ratio stabilizes.

Loom effect by environment

Effect	Perf-lab	Local
S4-JVM nvol/req	22.2	0.41
S4-Virtual nvol/req	2.75	1.34
S4 Loom TPS benefit	+20.5% (9,179 vs 7,616)	+12.1% (11,103 vs 9,901)
Q3 Loom TPS cost	-7.7% (12,241 vs 13,265)	-4.1% (15,041 vs 15,690)

The Loom benefit for Spring is larger on the perf-lab (+20.5% vs +12.1%). The Agroal experiment explains the difference: ~5% of the extra perf-lab benefit is HikariCP yield-spin elimination (perf-lab Loom benefit drops from +20.5% to +15.9% with Agroal, while local stays at ~12%). The remaining ~4% difference (15.9% vs 12%) may be CFS vs EEVDF scheduling overhead for 103 threads.

The Loom cost for Quarkus is also larger on the perf-lab (-7.7% vs -4.1%). The cause of this difference is not known.

Perf-Lab Validation: CFS Scheduler (RHEL 9.6, 17 Apr 2026)

Environment

Machine: Intel Xeon Gold 5218 @ 2.3GHz base, 3.9GHz turbo (Cascade Lake)
OS: RHEL 9.6, kernel 5.14.0-570.60.1.el9_6 → CFS scheduler (pre-EEVDF)
CPU pinning: app=2,4,6,8 (NUMA 0), db=10,12,14 (NUMA 0), otel=16,18,20, load-gen=22,24,26
Network: host networking (no pasta)
DB connection pool: 50
Warmup: 2 min wrk, 30s sleep, then 30s measurement
Iterations: 3 per runtime
wrk: 2 threads, 100 connections

TPS

Runtime	iter 0	iter 1	iter 2	Avg
Q3-JVM	13,243	13,267	13,284	13,265
Q3-Virtual	12,227	12,303	12,192	12,241
S4-JVM	7,509	7,733	7,605	7,616
S4-Virtual	9,216	9,210	9,112	9,179

Latency (avg)

Runtime	iter 0	iter 1	iter 2
Q3-JVM	7.54ms	7.52ms	7.51ms
Q3-Virtual	8.16ms	8.11ms	8.19ms
S4-JVM	13.30ms	12.91ms	13.13ms
S4-Virtual	10.82ms	10.84ms	10.95ms

Thread model (pidstat, during measurement window, iter 0)

Runtime	Poller/Event-loop	Worker model	Active workers
Q3-JVM	4 vert.x-eventloop	51 executor-thread (platform)	~51
Q3-Virtual	4 vert.x-eventloop + MasterPoller	4 FJ carriers + delayScheduler	~10 OS threads
S4-JVM	1 http-nio-8080-Poller	108 http-nio-8080-exec (platform)	~108
S4-Virtual	1 http-nio-8080-Poller + MasterPoller	4 FJ carriers + delayScheduler	~7 OS threads

Context switches (all JVM threads, iter 0, 20s measurement window)

Same methodology as local extraction: 20s sub-window from mpstat, context switches summed across all JVM threads, %wait averaged across worker threads only.

Runtime	TPS	Vol/s	Nvol/s	Vol/req	Nvol/req	Worker %wait	EL/Poller %wait
Q3-JVM	13,265	41,020	5,977	3.09	0.45	34.0%	87.4%
Q3-Virtual	12,241	39,477	39,033	3.23	3.19	17.0%	0.8%
S4-JVM	7,616	24,134	169,184	3.17	22.2	28.5%	65.2%
S4-Virtual	9,179	19,189	25,258	2.09	2.75	5.7%	0.3%

CPU utilization (mpstat, measurement window, iter 0)

Resource	Q3-JVM	Q3-Virtual	S4-JVM	S4-Virtual
App CPUs (2,4,6,8) busy	99.8%	90.1%	91.1%	89.2%
App %usr	77.2%	68.9%	70.7%	70.3%
App %sys	16.6%	16.1%	16.0%	14.2%
DB CPUs (10,12,14) busy	49.0%	37.7%	24.3%	28.3%

Observations

Q3-JVM is the only CPU-saturated runtime

Q3-JVM reaches 99.8% app CPU utilization (from mpstat above) while the other three are at 89-91%. The cause of the ~10% idle headroom in the other runtimes is not known.

Q3-JVM thread count: 101 (local) vs 51 (perf-lab)

Q3-JVM runs 101 executor-thread workers locally but only 51 on the perf-lab. Both are CPU-saturated (~100% app busy). Event-loop %wait: 75.4% (local) vs 87.4% (perf-lab). Postgres backend data shows only 31 out of 50 connections active on the perf-lab (vs 50 locally). The cause of the thread count difference is not known.

Cross-Environment Comparison

Full comparison table

Metric	Local (kernel 6.19)	Perf-lab (kernel 5.14)	Ratio
S4-JVM nvol/req	0.41	22.2	54x
S4-Virtual nvol/req	1.34	2.75	2.1x
S4-JVM %wait	48.9%	28.5%
S4-Virtual %wait	7.4%	5.7%
Q3-JVM nvol/req	0.08	0.45	6x
Q3-Virtual nvol/req	1.40	3.19	2.3x
Q3-JVM vol/req	3.05	3.09	~same
Q3-Virtual vol/req	1.60	3.23	2x
S4-JVM vol/req	2.34	3.17	1.4x
S4-Virtual vol/req	1.43	2.09	1.5x

Observations

S4-JVM nvol/req is 54x higher on the perf-lab (22.2 vs 0.41). The cause is not known — the environments differ in kernel version, CPU microarchitecture, and other factors. We cannot isolate any single variable.

S4-JVM %wait is higher locally (48.9% vs 28.5%) despite higher TPS. %wait measures time a thread is runnable but waiting in the queue for a CPU.

HikariCP ConcurrentBag contention (S4-JVM only)

HikariCP ConcurrentBag.requite() dominates S4-JVM context switches on the perf-lab and is a significant CPU cost difference between environments.

Context switch profile (from async-profiler -e context-switches, S4-JVM):

	Perf-lab	Local	Ratio
ConcurrentBag total	96,903 (48.4% of all cswitch)	13,979 (12.0%)	6.9x
requite → Thread.yield → sched_yield	83,966	11,612	7.2x
requite → LockSupport.parkNanos → futex	7,959	958	8.3x
borrow (scanning/waiting)	4,869	1,513	3.2x

CPU profile (from async-profiler -e cpu, S4-JVM):

	Perf-lab	Local	Ratio
ConcurrentBag CPU samples	229 / 3,791 (6.0%)	10 / 3,831 (0.26%)	23x
→ requite	200	7	29x
→ borrow	29	3	10x

Source code analysis (HikariCP 7.0.2, ConcurrentBag.requite()):

public void requite(final T bagEntry) {
   bagEntry.setState(STATE_NOT_IN_USE);
   for (int i = 1, waiting = waiters.get(); waiting > 0; i++, waiting = waiters.get()) {
      if (bagEntry.getState() != STATE_NOT_IN_USE || handoffQueue.offer(bagEntry)) {
         return;  // someone took it or handoff succeeded
      }
      else if ((i & 0xff) == 0xff || (waiting > 1 && i % waiting == 0)) {
         parkNanos(MICROSECONDS.toNanos(10));  // park every 256th or every waiting-th iteration
      }
      else {
         Thread.yield();  // default: yield and retry
      }
   }
   // if no waiters, add to threadLocal list
}

Thread.yield() fires when all of these are true:

waiters > 0 — at least one thread is inside borrow() waiting for a connection
handoffQueue.offer(bagEntry) fails — the SynchronousQueue has no matching waiter in poll() yet
bagEntry is still STATE_NOT_IN_USE — nobody else claimed it
Not on a park iteration (every 256th, or every waiting-th when waiting > 1)

In borrow(), waiters is incremented before scanning the shared list and entering poll(). There is a gap between waiters++ and entering handoffQueue.poll() where offer() will fail even though a waiter exists. How long threads spend in that gap — and how many are simultaneously in it — determines how many yield iterations each requite() call spins through.

The yield:park ratio is similar on both environments (~11:1), consistent with the park condition being purely iteration-count-based. What differs is the total number of spin iterations per requite call — 7x more on the perf-lab.

We don't know what causes requite() to spin more on the perf-lab. One hypothesis was postgres round-trip time — tested and ruled out:

Postgres RTT is faster on the perf-lab (confirmed by pgbench — see below). Faster DB means connections are returned sooner, so there should be less contention in requite, not 7x more. The contention must be driven by how 108 Tomcat threads interact with the CFS scheduler on 4 cores (vs Quarkus's 51 threads with Agroal, which does not use a yield-spin handoff).

Postgres RTT measurement

Measured with pgbench: single connection, prepared statements, 10 seconds, pinned to app CPUs. Both environments use host networking and containerized postgres 17.9.

Test 1: SELECT 1 (pure RTT, no result set)

pgbench-rtt.sql contains SELECT 1;.

	Local	Perf-lab
Latency avg	0.041 ms	0.027 ms
TPS	24,447	36,903
Ratio	1.5x slower	1x

Test 2: Actual app queries (2 serial queries per transaction)

pgbench-rtt.sql contains the 2 queries the app executes per request:

SELECT id, name, description FROM fruits;
SELECT store_id, fruit_id, price FROM store_fruit_prices WHERE fruit_id IN (1,2,3,4,5,6,7,8,9,10);

The first query returns 10 rows (all fruits). The second returns 32 rows (all store prices for those fruits). Both execute serially, causing 2 RTTs per transaction — matching what Hibernate does per GET /fruits request.

	Local (10 runs)	Perf-lab
Latency avg	0.109–0.116 ms (median 0.113)	0.093 ms
TPS	8,604–9,146	10,710
Ratio	1.2x slower	1x

The gap narrows from 1.5x (pure RTT) to 1.2x (with result sets). The result set transfer cost (42 rows total) is similar on both environments, so it dilutes the pure RTT difference.

The perf-lab postgres RTT is faster than local on both tests.

Impact on open questions

Q3-JVM thread count (101 local vs 51 perf-lab): RTT is consistent — faster DB means each thread blocks less, so fewer threads needed to saturate 4 cores. But 1.2x RTT difference doesn't quantitatively explain a 2x thread count difference. The JBoss Threads pool growth algorithm may amplify a small difference (fewer tasks queued → pool doesn't grow as aggressively). Not fully explained.

HikariCP 7x contention: RTT deepens the mystery. Faster DB means connections return sooner → requite() should spin less, not 7x more. Rules out DB latency as the cause. The contention must come from the interaction of 108 threads with the CFS scheduler on 4 cores.

S4-JVM %wait (48.9% local vs 28.5% perf-lab): Not explained by RTT. %wait measures CPU queue time (runnable but not running), not I/O wait. This is about scheduler behavior with 108 threads competing for 4 cores.

Both runtimes lose ~2.3K TPS local→perf-lab: RTT cannot explain this — the DB is faster on perf-lab, so it should add TPS, not subtract. The shared loss must come from a non-DB factor (kernel overhead, microarchitecture).

Poller/event-loop %wait much higher locally (S4-Virtual 30.7% vs 0.3%, Q3-Virtual 14.7% vs 0.8%): Consistent with RTT data — slower DB locally means workers block longer, pollers/event-loops spend more time idle waiting for workers to produce responses.

Q3-JVM event-loop %wait (75.4% local vs 87.4% perf-lab): Consistent — faster DB on perf-lab means workers complete faster, event loops wait more between bursts of response writes.

Quarkus vs Spring gap

Pair	Local (HikariCP)	Perf-lab (HikariCP)	Local (Agroal)	Perf-lab (Agroal)
Q3-JVM / S4-JVM	1.58x	1.74x	1.62x	1.63x
Q3-JVM / S4-Virtual	1.41x	1.44x	—	—
Q3-Virtual / S4-Virtual	1.35x	1.33x	—	—

The original investigation observed a 1.87x Q/S ratio on the perf-lab (with busy_poll=50, broken OTel sampling). After fixing those: 1.74x (HikariCP, host networking). HikariCP contention accounts for the remaining environment difference: with Agroal, the ratio stabilizes at ~1.63x on both environments — this is the intrinsic per-request CPU cost difference between Quarkus and Spring (framework overhead, not pool contention). Once both use Loom, the HikariCP ratio also stabilizes (~1.34x).

Poller/event-loop %wait

Runtime	Poller/EL %wait (local)	Poller/EL %wait (perf-lab)
S4-Virtual	30.7%	0.3%
Q3-Virtual	14.7%	0.8%

Pollers and event-loops wait much longer locally than on the perf-lab. The cause is not known.

Q3-JVM thread count: 101 (local) vs 51 (perf-lab)

Q3-JVM uses 101 executor threads locally but only 51 on the perf-lab. The pool max is 200 on both. Both reach 99.8% app CPU. The cause of this difference is not known — it is one of two key unexplained observations (the other being the different context switch distributions).

Active postgres backends during measurement (pidstat >1% CPU, 30s window)

	Q3-JVM	S4-JVM
Local	50	27
Perf-lab	38	19
DB pool size	50	50
App threads	101 (local) / 51 (perf-lab)	108 (both)
TPS	15,690 (local) / 13,265 (perf-lab)	9,901 (local) / 7,616 (perf-lab)

Data sources: local Q3-JVM from local-loom-comparison/, local S4-JVM from scaling-others-c100/, perf-lab from spring-vs-quarkus-17apr-lab/. All runs: 100 wrk connections, 4 app cores, host networking. Measurement window identified from wrk 30s start timestamp in run.log; perf-lab pidstat uses 12h AM/PM format (server timezone differs from run.log UTC offset).

Spring consistently uses fewer active backends than Quarkus on both environments. On the perf-lab, both use fewer than locally. The cause is not known.

Experiment: Spring Boot 4 with Agroal instead of HikariCP

Branch spring_w_agroal (commit bad1e31): replaced HikariCP with Agroal standalone (io.agroal:agroal-pool 3.0.1) in Spring Boot 4, configured to match Quarkus defaults (maxSize=50, minSize=0, acquisitionTimeout=5s, validationTimeout=2m, reapTimeout=5m). The goal is to test whether HikariCP's yield-spin contention is responsible for Spring's worse performance on the perf-lab.

TPS

	HikariCP	Agroal	Change
Perf-lab	7,616	8,132	+6.8%
Local	9,901	~9,400	~same

Perf-lab: scheduling and contention

Metric	HikariCP	Agroal	Change
nvol/req	22.2	0.23	-96x
vol/req	3.17	1.99	-37%
%wait (Tomcat threads)	28.5%	13.5%	-53%
Thread.yield CPU samples	200 / 3,791 (5.3%)	0 / 3,948	gone
Active postgres backends (>1% CPU)	19	50	+163%
Tomcat threads	108	103	same

HikariCP's yield-spin is completely eliminated with Agroal: zero Thread.yield / sched_yield samples in the JFR flamegraph. Involuntary preemptions per request drop 96x. All 50 DB connections are now actively utilized (vs only 19 with HikariCP). %wait drops by half.

Locally, TPS is unchanged — consistent with HikariCP contention being negligible on EEVDF (only 10 CPU samples locally vs 229 on perf-lab).

Interpretation

HikariCP's ConcurrentBag.requite() yield-spin accounts for ~500 TPS of the S4-JVM gap on the perf-lab (7,616 → 8,132). The remaining gap to Q3-JVM (8,132 vs 13,265 = 1.63x) is per-request CPU cost in Spring MVC, Tomcat, and Hibernate — not connection pool contention.

The 19 → 50 active backends confirms the mechanism: with HikariCP, the yield-spin handoff was blocking connections in a "returned but not yet handed off" state, preventing other threads from using them. Agroal doesn't have this handoff pattern.

S4-Virtual (Loom) with Agroal

	Local HikariCP	Local Agroal	PL HikariCP	PL Agroal
S4-JVM TPS	9,901	9,644	7,616	8,132
S4-Virtual TPS	11,103	10,792	9,179	9,425
Loom benefit	+12.1%	+11.9%	+20.5%	+15.9%

All local data collected with JDK 25.0.2-tem on branch spring_w_agroal. Perf-lab data from Jenkins runs #301 (JVM) and #303 (Virtual).

Locally, the Loom benefit is the same (~12%) regardless of pool. Agroal is ~3% slower than HikariCP on both JVM and Virtual — a small constant overhead in Agroal's borrow/return path vs HikariCP's optimized ConcurrentBag fast-path (when contention is low, as it is locally on EEVDF).

On the perf-lab, the Loom benefit drops from +20.5% (HikariCP) to +15.9% (Agroal). The ~5% difference is the HikariCP yield-spin elimination that Loom provided — gone with Agroal since there's no yield-spin to eliminate.

Loom benefit summary:

Pool	Local Loom benefit	Perf-lab Loom benefit
HikariCP	+12.1% (9,901 → 11,103)	+20.5% (7,616 → 9,179)
Agroal	+11.9% (9,644 → 10,792)	+15.9% (8,132 → 9,425)

Locally, Loom helps equally with both pools (~12%). The benefit is from OS thread count reduction (103 → ~7), not from pool contention elimination.

On the perf-lab, Loom helps more with HikariCP (+20.5% vs +15.9%) because it eliminates two sources of waste: (1) the yield-spin contention and (2) CFS scheduling overhead from 103 threads. With Agroal, only (2) remains.

Loom is not always an improvement. When the pool and scheduler already work efficiently, Loom's overhead (virtual thread mount/unmount, ForkJoinPool scheduling) may produce no gain. In this workload, the ~12% local benefit comes from reducing 103 OS threads to ~7 even on EEVDF — but this benefit happens to offset the VT overhead almost exactly. On a workload with less blocking (shorter DB queries, fewer syscalls), Loom's overhead might exceed its benefit.

Data: perf-lab S4-JVM+Agroal at /home/forked_franz/Documents/spring-vs-quarkus-27apr/perf lab spring 4 w Agroal/, perf-lab S4-Virtual+Agroal at /home/forked_franz/Documents/spring-vs-quarkus-27apr/perf lab spring 4 w Agroal + Loom/, local Agroal (JDK 25) at /tmp/spring4-agroal-correct/.

Confounding variables

The two environments differ in multiple dimensions simultaneously:

CPU: Intel Xeon Cascade Lake vs AMD Zen 4 — different IPC, cache, branch prediction (both locked to 2.3GHz)
Kernel: 5.14 vs 6.19 — different scheduler (CFS vs EEVDF), network stack, syscall implementation
NUMA: both environments pin app and DB to the same NUMA node — NUMA is not a factor
CPU vulnerability mitigations: differ significantly (see table below)
tuned profile: perf-lab uses custom benchmark profile (inherits latency-performance, disables turbo, sets busy_poll=0). Does NOT override CFS scheduler tunables — sched_min_granularity_ns=3000000 (3ms) is the kernel default scaled 4x for 64 CPUs.

CPU vulnerability mitigations (`/sys/devices/system/cpu/vulnerabilities/`)

Vulnerability	Local (AMD Zen 4)	Perf-lab (Intel Xeon Cascade Lake)
gather_data_sampling	Not affected	Mitigation: Microcode
indirect_target_selection	Not affected	Mitigation: Aligned branch/return thunks
itlb_multihit	Not affected	KVM: Mitigation: Split huge pages
l1tf	Not affected	Not affected
mds	Not affected	Not affected
meltdown	Not affected	Not affected
mmio_stale_data	Not affected	Mitigation: Clear CPU buffers; SMT vulnerable
retbleed	Not affected	Mitigation: Enhanced IBRS
spec_rstack_overflow	Mitigation: Safe RET	Not affected
spec_store_bypass	SSBD disabled via prctl	SSBD disabled via prctl
spectre_v1	usercopy/swapgs barriers	usercopy/swapgs barriers
spectre_v2	Enhanced IBRS; IBPB conditional; STIBP always-on	Enhanced IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS: SW sequence; BHI: SW loop
srbds	Not affected	Not affected
tsa	Mitigation: Clear CPU buffers	(not reported — older kernel)
tsx_async_abort	Not affected	Mitigation: TSX disabled
vmscape	Mitigation: IBPB before exit to userspace	(not reported — older kernel)

STIBP context switch cost

The key mitigation difference is STIBP (Single Thread Indirect Branch Predictor). On AMD Ryzen, the kernel sets STIBP: always-on because AMD's SMT requires explicit cross-thread indirect branch isolation that eIBRS alone doesn't provide. On Intel Cascade Lake, eIBRS provides that isolation in hardware — no STIBP needed.

STIBP forces a write to the IA32_SPEC_CTRL MSR on every context switch — a fully serializing operation.

Context switch microbenchmark (perf bench sched pipe):

Scenario	Local Ryzen (STIBP always-on)	Perf-lab Xeon (no STIBP)	Ratio
Single CPU (`taskset -c 0`)	9.37 µs/op (106K ops/s)	3.76 µs/op (266K ops/s)	2.5x
2 physical cores	4.74 µs/op (211K ops/s)	2.14 µs/op (466K ops/s)	2.2x
Single CPU + forced STIBP (perf-lab)	—	10.54 µs/op (94K ops/s)

Forcing STIBP on the perf-lab via prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE) dropped it to 94K ops/s — below the local Ryzen baseline. This confirms STIBP is the dominant per-switch cost difference.

STIBP experiment (RHEL 10, 5 May 2026)

Forced STIBP on the RHEL 10 perf-lab (kernel 6.12, EEVDF) via sudo grubby --update-kernel=ALL --args="spectre_v2_user=on" + reboot. Verified active: SpeculationIndirectBranch: always disabled in /proc/self/status. perf bench sched pipe dropped from ~266K to 93.5K ops/s on single CPU — matching the prctl experiment, confirming STIBP is active.

Note: spectre_v2_user=on also forces IBPB: always-on (local AMD has IBPB: conditional). No boot parameter isolates STIBP alone.

Runtime	RHEL 10 + STIBP	RHEL 9.6 (no STIBP)	Local (STIBP)
Q3-JVM	12,924	13,265	15,690
Q3-Virtual	12,645	12,241	15,041
S4-JVM	7,284	7,616	9,901
Q3-JVM / S4-JVM ratio	1.77x	1.74x	1.58x

The ratio did not converge toward the local 1.58x. STIBP is not the driver of the ratio difference between environments.

S4-JVM context switches (1 second during measurement window, all JVM threads):

Metric	RHEL 10 + STIBP (EEVDF)	RHEL 9.6 (CFS, no STIBP)	Local (EEVDF, STIBP)
vol/s	36,414	24,134	23,140
nvol/s	10,622	169,184	4,055
vol/req	5.0	3.17	2.34
nvol/req	1.46	22.2	0.41

The nvol storm (22.2/req on CFS) is gone on EEVDF (1.46/req) — this is an EEVDF effect, not STIBP. But vol/req is unexpectedly high (5.0) compared to both other environments. The cause is not known — HikariCP yield-spin or another source of voluntary switches could be involved, but no profiling data was collected to confirm.

Rollback: sudo grubby --update-kernel=ALL --remove-args="spectre_v2_user=on" + reboot.

Data: perf-lab run output at /home/jenkins/agent/workspace/Products/Quarkus/Competitors/spring-quarkus-perf-comparison/run/ on mwperf-server14.

sched_yield microbenchmark

perf bench sched pipe measures full context switch cost (pipe read/write forces a reschedule). HikariCP's ConcurrentBag.requite() uses Thread.yield() → sched_yield, which goes through a different scheduler path. To isolate yield cost, two microbenchmarks were written:

C benchmark (/tmp/yield_bench.c): two pthreads pinned to one CPU, both calling sched_yield() in a tight loop for 5 seconds.
Java benchmark (/tmp/YieldBench.java): N platform threads pinned to one CPU, all calling Thread.yield() in a tight loop for 5 seconds, with N ∈ {2, 10, 50, 100}.

Java Thread.yield() maps directly to sched_yield — C and Java benchmarks produce identical results at 2 threads (2.26 vs 2.28 us local, 0.89 vs 0.91 us perf-lab).

C results (2 threads, pinned to 1 CPU):

Environment	us/yield	yields/s
Local (AMD Zen 4, EEVDF, STIBP, kernel 6.19)	2.26	444K
Perf-lab (Intel Cascade Lake, EEVDF, STIBP, kernel 6.12)	0.89	1.12M
Ratio (local / perf-lab)	2.5x

Java results (Thread.yield(), pinned to 1 CPU, varying thread count):

Threads	Local (AMD) us/yield	Perf-lab (Intel) us/yield	Ratio
2	2.28	0.91	2.5x
10	2.38	0.95	2.5x
50	2.49	1.07	2.3x
100	2.59	1.17	2.2x

Thread count scaling: +13% (local) and +28% (perf-lab) from 2→100 threads. The scheduler's pick-next decision gets more expensive with more runnable threads, and Intel's advantage narrows slightly at higher counts (2.5x → 2.2x).

Intel is 2.2–2.5x faster at sched_yield than AMD, both with STIBP enabled on EEVDF. This means HikariCP's yield-spin per-iteration cost is lower on the perf-lab, not higher. The high vol/req on RHEL 10 (5.0 vs 2.34 local) cannot be explained by yield being more expensive — it must be driven by more yield iterations per requite call.

#define _GNU_SOURCE
#include <sched.h>
#include <time.h>
#include <stdio.h>
#include <pthread.h>
#include <stdlib.h>

static volatile int running = 1;
static long count1 = 0, count2 = 0;

void *yielder(void *arg) {
    long *count = (long *)arg;
    while (running) {
        sched_yield();
        (*count)++;
    }
    return NULL;
}

int main(int argc, char **argv) {
    int seconds = 5;
    pthread_t t1, t2;
    pthread_create(&t1, NULL, yielder, &count1);
    pthread_create(&t2, NULL, yielder, &count2);
    struct timespec ts = {seconds, 0};
    nanosleep(&ts, NULL);
    running = 0;
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    long total = count1 + count2;
    printf("%ld yields in %ds = %.3f us/yield (%ld yields/s)\n",
           total, seconds, (double)seconds * 1e6 / total, total / seconds);
    return 0;
}

RHEL 10 perf-lab results (prior to STIBP experiment)

RHEL 10 (kernel 6.12) on the perf-lab showed no performance difference vs RHEL 9.6 (kernel 5.14) for Spring, but Q3-JVM regressed from 13.3K to 12.6K TPS. Loom had no adverse effect on the RHEL 10 run (Q3-Virtual TPS same as Q3-JVM). Cause of the Q3-JVM regression is not known.

We cannot attribute observed differences to any single variable. A CPU profile comparison (with busy_poll=0) shows nearly identical per-request application work on both environments — the CPU does the same work per request, but TPS differs.

Benchmark Repos and Branches

Branch	Repo	Purpose
`main`	upstream / fork	Default benchmark (HikariCP), scalability tests, pgbench RTT test
`spring_w_agroal`	fork	Spring Boot 4 with Agroal instead of HikariCP (commit bad1e31)

Data Locations

Local (all under /home/forked_franz/Documents/spring-vs-quarkus-18apr/):
- Q3-JVM 1-conn (3 iter, with flamegraph): 1conn-experiment/
- Q3-Virtual 1-conn (3 iter, with flamegraph): 1conn-experiment-virtual/
- Q3-JVM scaling 2-100: scaling-jvm-c{2..100}/
- Others scaling 1-100: scaling-others-c{1..100}/
- 8-core host net: scaling-8core-host-c50/ (Q3-JVM, Q3-Virtual, S4-JVM from run log; S4-Virtual wrk files lost due to crash)
- 8-core host net S4-Virtual retry: scaling-8core-host-c50-s4vt-retry/
- 8-core pasta: scaling-8core-pasta-c50/
- Q3-JVM/Virtual syncjfr (lock investigation): lock-investigation/, lock-investigation-virtual/
- Q3-JVM/Virtual syncjfr rerun (27 Apr): /tmp/q3-syncjfr-rerun/
- S4-JVM/Virtual Agroal (JDK 25): /tmp/spring4-agroal-correct/
- This document: scalability-local-20apr-spring-vs-quarkus/
Perf-lab (CFS validation): /home/forked_franz/Documents/spring-vs-quarkus-17apr-lab/
Perf-lab (Agroal experiment): /home/forked_franz/Documents/spring-vs-quarkus-27apr/

Scheduler References

EEVDF (Linux 6.6+)

An EEVDF CPU scheduler for Linux — LWN, Mar 2023. Peter Zijlstra's proposal: pick_eevdf(), virtual deadline, lag-based eligibility.
Completing the EEVDF scheduler — LWN, Apr 2024. PREEMPT_SHORT, per-task slice via sched_setattr(), deferred dequeue.
Reports from OSPM 2024 — Google found stock EEVDF underperformed CFS on ChromeOS; quadrupling base slice restored parity; removing eligibility yielded 30% improvement.
sched/eevdf: Curb wakeup-preemption — The RUN_TO_PARITY commit. Reduced context switches 31% on EPYC 128-core.
EEVDF kernel docs — Official (brief).
kernel/sched/features.h — Feature flag defaults: RUN_TO_PARITY(true), PREEMPT_SHORT(true), DELAY_DEQUEUE(true).
Virginia Tech CS 5264 Lecture 07 — Detailed walk-throughs of pick_eevdf(), lag, deadline formula, protect_slice().

CFS (pre-6.6, kernel 5.14 on perf-lab)

CFS Scheduler Design — Canonical kernel docs.
kernel v5.14 fair.c — Source: check_preempt_tick(), check_preempt_wakeup(), __sched_period(), get_update_sysctl_factor().
SysTutorials: CFS Tunables — How sched_latency, sched_min_granularity, sched_wakeup_granularity interact.
CFS tunable scaling — num_online_cpus() scaling, SCHED_TUNABLESCALING_LOG.
RHEL 9 kernel config — CONFIG_HZ=1000, PREEMPT_VOLUNTARY (via PREEMPT_DYNAMIC).

Scheduler features (`/sys/kernel/debug/sched/features`)

Feature	Perf-lab (5.14)	Local (6.19)
`WAKEUP_PREEMPTION`	enabled	enabled
`RUN_TO_PARITY`	N/A	enabled
`GENTLE_FAIR_SLEEPERS`	enabled	N/A
`PLACE_LAG`	N/A	enabled
`PREEMPT_SHORT`	N/A	enabled
`DELAY_DEQUEUE`	N/A	enabled

These features differ between the two kernels, but we cannot attribute the observed performance differences to any specific feature.

Context switch overhead (reference)

Eli Bendersky: Measuring context switching — 1.0–1.5μs direct cost pinned, ~2.2μs cross-core.
Tsuna: How long does a context switch take? — 2.7–5.5μs with minor TLB effects.
Coding Confessions: Context Switching — Full cache/TLB cold start: 10–100+μs.
Coding Confessions: TLB Flushes — With PCID, no TLB flush for same-process thread switches on modern x86.

franz1981/unified_theory.md

Scalability Test: Spring vs Quarkus — Local, 20 Apr 2026

Environment

Runtimes

TPS Scaling Curve

Latency (avg)

Observations

1. Per-request baseline (1 connection, zero contention)

2. Quarkus scaling: JVM and Virtual converge at saturation

3. Spring: virtual threads OVERTAKE platform threads at high concurrency

4. Spring4-JVM degrades at high concurrency

5. Quarkus vs Spring gap widens at saturation

6. Plateau analysis

8-Core Test (50 connections, 21 Apr 2026)

TPS: Host Networking vs Pasta

Latency (avg, 50 connections)

4-core vs 8-core scaling (host networking, 50 connections)

Observations

7. Virtual threads overtake JVM threads at 8 cores

8. Sub-2x scaling from 4→8 cores

9. Pasta proxy is a hard ceiling

Scheduling Analysis: Why Loom Helps Spring but Not Quarkus (100 connections, 4 cores)

Thread model per runtime

Context switches (all JVM threads, 20s measurement window, 100 connections)

Interpretation

Loom effect by environment

Perf-Lab Validation: CFS Scheduler (RHEL 9.6, 17 Apr 2026)

Environment

TPS

Latency (avg)

Thread model (pidstat, during measurement window, iter 0)

Context switches (all JVM threads, iter 0, 20s measurement window)

CPU utilization (mpstat, measurement window, iter 0)

Observations

Q3-JVM is the only CPU-saturated runtime

Q3-JVM thread count: 101 (local) vs 51 (perf-lab)

Cross-Environment Comparison

Full comparison table

Observations

HikariCP ConcurrentBag contention (S4-JVM only)

Postgres RTT measurement

Impact on open questions

Quarkus vs Spring gap

Poller/event-loop %wait

Q3-JVM thread count: 101 (local) vs 51 (perf-lab)

Active postgres backends during measurement (pidstat >1% CPU, 30s window)

Experiment: Spring Boot 4 with Agroal instead of HikariCP

TPS

Perf-lab: scheduling and contention

Interpretation

S4-Virtual (Loom) with Agroal

Confounding variables

CPU vulnerability mitigations (/sys/devices/system/cpu/vulnerabilities/)

STIBP context switch cost

STIBP experiment (RHEL 10, 5 May 2026)

sched_yield microbenchmark

RHEL 10 perf-lab results (prior to STIBP experiment)

Benchmark Repos and Branches

Data Locations

Scheduler References

EEVDF (Linux 6.6+)

CFS (pre-6.6, kernel 5.14 on perf-lab)

Scheduler features (/sys/kernel/debug/sched/features)

Context switch overhead (reference)

CPU vulnerability mitigations (`/sys/devices/system/cpu/vulnerabilities/`)

Scheduler features (`/sys/kernel/debug/sched/features`)