| elastic01 | elastic45 | |
|---|---|---|
| CPU | AMD Ryzen 9 5950X — 16 cores / 32 threads | 96 processors (likely dual EPYC) |
| CPU usage | 93% | 48% |
| Load average | 66 / 62 / 55 (2× core count!) | Not measured, but much lower |
| ES allocated_processors | 32 | 96 |
This is the dominant factor. elastic01 is severely CPU-constrained. The search thread pool is saturated at 49 active threads with 86 requests queued, while elastic45 has 0 queued. With 3× the cores, elastic45 handles the same workload at half the CPU utilization.
| elastic01 | elastic45 | |
|---|---|---|
| Search active | 49 | 43 |
| Search queue | 86 🔴 | 0 ✅ |
| Search rejected | 7,439,017 | 12,762,625 |
| Search completed | 14.9B | 63.8B (4× more throughput) |
| Search query total | 842M | 3,478M (4× more queries) |
elastic01's search queue is backing up because the CPU can't keep up with the thread pool. elastic45 processes 4× more search queries with no queue buildup.
| Layer | Detail |
|---|---|
| Physical | 2× Samsung PM9A3 3.84TB NVMe (MZQL23T8HCLS-00A07) |
| RAID | md2 = RAID0 (512K chunks) across nvme0n1p3 + nvme1n1p3 |
| Encryption | LUKS2 on top of RAID0, cipher aes-xts-plain64, key 256-bit, PBKDF argon2id |
| Filesystem | ext4 with stride=128, stripe-width=512 (tuned for RAID0) |
| Discard | allow_discards enabled, discard in crypttab, fstrim.timer running |
| Readahead | 256 (128KB) on all devices |
| I/O scheduler | none on NVMe, mq-deadline available |
LUKS overhead is significant: During peak, dm-0 (the LUKS device) hits 97.6% utilization while the underlying NVMe drives are only at ~45%. The aes-xts-plain64 encryption/decryption adds CPU overhead — system CPU is at 24% (encryption + kernel), which eats into the already-limited 32 cores.
| Device | %util | r_await | w_await | r/s | rkB/s |
|---|---|---|---|---|---|
| dm-0 (LUKS) | 97.6% | 0.47ms | 0.48ms | 20,652 | 1,904,872 |
| md2 (RAID0) | 81.2% | 0.14ms | 0.06ms | 30,841 | 1,904,988 |
| nvme0n1 | 45.5% | 0.13ms | 0.03ms | 14,909 | 954,320 |
| nvme1n1 | 45.6% | 0.12ms | 0.04ms | 15,746 | 949,884 |
The dm-0 bottleneck (97.6%) vs NVMe (45%) shows the LUKS layer is the I/O constraint. On a 96-core machine, the encryption CPU overhead is less impactful.
| elastic01 | elastic45 | |
|---|---|---|
| Total shards | 119 (65P + 54R) | 101 (61P + 40R) |
| Doc count | 1,373M | 1,663M |
| Store size | 3,418 GB | 3,909 GB |
Shard counts are comparable — not a contributing factor.
| elastic01 (up 168h) | elastic45 (up 511h) | |
|---|---|---|
| Heap used | 50% | 41% |
| Heap max | 26 GB | 26 GB |
| GC young collections | 69,892 | 351,101 |
| GC young time/collection | 92.5ms | 35.2ms |
elastic01's GC pauses are 2.6× longer per collection, again because the CPU is saturated.
Both nodes run identical ES config: path.data=/data, same heap (26G), same image (8.19.16), same thread pool defaults (search.size=49, queue=1000).
-
🔴 CPU count (32 vs 96 cores) — The primary bottleneck. elastic01's Ryzen 9 5950X is simply 3× weaker than whatever dual-EPYC machine elastic45 sits on. This directly limits search throughput and causes queue buildup.
-
🟡 LUKS encryption overhead —
aes-xts-plain64on a 32-core machine eats ~24% system CPU. On elastic45's 96 cores, the same encryption overhead is diluted across 3× the cores. The dm-0 device hits 97.6% utilization on elastic01. -
🟡 Search queue saturation — With 86 queued searches and 7.4M rejected, elastic01 is dropping/throttling requests. This cascades into higher latency for all queries hitting that node.
-
🟢 Disk I/O is NOT the bottleneck — The underlying NVMe drives are only at 45% utilization with sub-0.2ms latency. The RAID0+LUKS stack is the constraint, not raw disk speed.
- Rebalance shards away from elastic01 if possible (it has 119 vs elastic45's 101, but the difference is minor)
- Reduce LUKS overhead: Switch to
aes-xts-plain64with hardware AES-NI (already using it), or consider removing LUKS if security allows - Upgrade elastic01's hardware to match the 96-core nodes
- Tune
thread_pool.search.queue_sizeor add circuit breakers to fail fast rather than queue