6-Way Comparison with Compression Variants
MinIO vs SeaweedFS vs Garage vs Garage-zstd vs HS5 vs RustFS
Date: 2026-03-25
Method: Python boto3 direct, sub-ms accuracy
Data: 746 real TDnet filing documents, 600.2 MB
Each backend tested in isolation — fresh volumes, one at a time, no resource contention
| Backend | Compression | Durability | Volumes | Notes |
|---|---|---|---|---|
| MinIO | None | fsync per write | 1 | Fresh instance on port 9100 |
| SeaweedFS | Always on (content-type based) | Default | 1 volume, 3 containers | Cannot disable — compresses HTML/JSON, skips PNG/JPG |
| Garage | None (compression_level = 'none') |
No fsync | 1 | metadata_fsync=false, data_fsync=false, disable_scrub=true |
| Garage-zstd | zstd level 3 (compression_level = 0) |
No fsync | 1 | Same as above but with compression |
| HS5 | None | Manual commit only | 1 | --manual-commit --wal-mode full |
| RustFS | None | Default | 1 (single volume) | Fixed from Round 2 (was 4-volume erasure coding) |
Round 2 fairness fixes applied:
- RustFS: 4 volumes → 1 volume (eliminated 2× erasure coding overhead)
- Garage:
compression_level=0was actually zstd level 3, not disabled — now split into two variants- SeaweedFS:
-compressUpload=falseflag doesn't exist — SeaweedFS always compresses compressible content
| Test | Result | Latency |
|---|---|---|
| Sequential Write 1MB (fdatasync) | 123 MB/s | — |
| Sequential Write 1MB (no sync) | 1,630 MB/s | — |
| Random Read 4K | 376,200 IOPS | p50=2μs |
| Random Write 4K (fsync) | 395 IOPS | p50=2.3ms |
| Backend | Time | MB/s | ops/s | CPU avg | Mem avg |
|---|---|---|---|---|---|
| RustFS | 1.6s | 370 | 460 | — | — |
| HS5 | 1.7s | 358 | 445 | — | — |
| Garage | 1.8s | 338 | 420 | — | — |
| Garage-zstd | 1.9s | 309 | 384 | — | — |
| SeaweedFS | 2.8s | 213 | 265 | — | — |
| MinIO | 3.0s | 199 | 247 | 90% | 385 MB |
RustFS with single volume is now competitive — 370 MB/s vs 328 MB/s in Round 2 (4-volume). HS5 close behind. Garage-zstd is ~9% slower than uncompressed Garage due to zstd CPU overhead.
| Backend | Disk Used | Ratio | Notes |
|---|---|---|---|
| Garage-zstd | 120 MB | 0.20× | zstd level 3 compresses filings 5:1. Best disk efficiency. |
| SeaweedFS | ~220 MB* | ~0.36×* | Always compresses. (*Disk measurement showed 0 due to volume container du artifact — using Round 2 value.) |
| Garage | 608 MB | 1.01× | No compression. Near 1:1. |
| MinIO | 611 MB | 1.02× | No compression. Near 1:1. |
| RustFS | 611 MB | 1.02× | Fixed! Single volume = 1:1. Was 2.07× with 4-volume erasure coding. |
| HS5 | 1030 MB | 1.72× | Data file pre-allocates to 1024 MB (next power of 2 for 600 MB data). |
Compression impact on Garage: 608 MB → 120 MB = 5.1× compression ratio on real filing data (HTML, JSON, text). The zstd overhead is only 9% slower upload speed — massive space savings for minimal performance cost.
Why Round 1 showed 1.29× for Garage but Round 3 shows 0.20×: Round 1 (FINAL_REPORT.md) used
os.urandom()— incompressible random bytes. zstd cannot compress random data, so the only overhead was LMDB metadata (1.29×). Round 3 uses real TDnet filings — HTML reports, JSON metadata, text-based XML — which zstd compresses excellently at 5:1. This demonstrates that compression benchmarks are meaningless with synthetic random data; real-world content must be used.
| Backend | CPU | Memory (docker stats) | Memory (cgroup) | Containers |
|---|---|---|---|---|
| Garage-zstd | 0.5% | 13 MB | ~25 MB† | 1 |
| Garage | 0.5% | 53 MB | ~90 MB† | 1 |
| HS5 | 0.3% | 69 MB | ~69 MB | 1 |
| RustFS | 0.4% | 345 MB | ~345 MB | 1 |
| MinIO | 0.7% | 424 MB | ~424 MB | 1 |
| SeaweedFS | 0.1% | 804 MB | ~804 MB | 3 |
† Garage memory caveat:
docker statsreports only anonymous RSS (heap). Garage uses LMDB withmmap(), which maps the database file into virtual memory. These mmap'd pages show up in cgroupmemory.currentbut not indocker stats. Measured: docker stats = 10.8 MB vs cgroup = 25.4 MB (2.3× under-reported) at idle with empty data. With data loaded, the gap is larger (~53 MB reported vs ~90 MB cgroup). The mmap'd pages are evictable under memory pressure — the OS treats them like page cache — so Garage is still the most memory-efficient backend, but not as dramatically asdocker statssuggests.
| Backend | ops/s | MB/s | p50 | p95 | CPU | Mem |
|---|---|---|---|---|---|---|
| HS5 | 616 | 518 | 1.4ms | 2.3ms | 33% | 73 MB |
| MinIO | 447 | 368 | 1.8ms | 3.9ms | 59% | 472 MB |
| RustFS | 399 | 315 | 2.1ms | 4.5ms | 54% | 613 MB |
| Garage | 264 | 208 | 2.2ms | 9.6ms | 77% | 55 MB |
| SeaweedFS | 219 | 180 | 2.2ms | 13.3ms | 102% | 816 MB |
| Garage-zstd | 193 | 164 | 2.6ms | 21.4ms | 86% | 15 MB |
HS5 dominates sequential reads: 616 ops/s at 33% CPU, 73 MB RAM. Garage-zstd is slowest due to decompression overhead on every read.
| Backend | ops/s | MB/s | p50 | p95 | CPU | Mem |
|---|---|---|---|---|---|---|
| HS5 | 212 | 194 | 2.4ms | 11.7ms | 59% | 136 MB |
| Garage | 193 | 163 | 3.0ms | 12.5ms | 131% | 43 MB |
| RustFS | 180 | 141 | 3.4ms | 13.6ms | 79% | 728 MB |
| Garage-zstd | 163 | 127 | 4.1ms | 14.0ms | 160% | 17 MB |
| SeaweedFS | 74 | 68 | 5.1ms | 47.8ms | 143% | 648 MB |
| MinIO | 74 | 62 | 13.2ms | 23.3ms | 28% | 493 MB |
HS5 is 2.9× faster than MinIO at sequential writes. MinIO's p50 of 13.2ms reflects its per-write fsync. Garage-zstd's compression adds ~15% CPU overhead vs uncompressed Garage.
| Backend | ops/s | MB/s | p50 | p95 | CPU | Mem |
|---|---|---|---|---|---|---|
| HS5 | 632 | 516 | 6.7ms | 16.9ms | 32% | 87 MB |
| Garage | 616 | 460 | 6.6ms | 16.7ms | 193% | 44 MB |
| MinIO | 597 | 482 | 7.3ms | 16.7ms | 79% | 472 MB |
| RustFS | 586 | 483 | 7.4ms | 17.6ms | 86% | 674 MB |
| SeaweedFS | 577 | 466 | 6.4ms | 19.1ms | 256% | 548 MB |
| Garage-zstd | 564 | 449 | 6.2ms | 26.3ms | 270% | 16 MB |
All converge at 5 workers (564–632 ops/s). HS5 uses 6× less CPU than Garage and SeaweedFS.
| Backend | ops/s | MB/s | p50 | p95 | CPU | Mem |
|---|---|---|---|---|---|---|
| Garage | 504 | 391 | 8.0ms | 18.6ms | 330% | 46 MB |
| RustFS | 503 | 385 | 7.8ms | 21.2ms | 235% | 868 MB |
| Garage-zstd | 452 | 368 | 9.0ms | 21.8ms | 492% | 21 MB |
| HS5 | 415 | 328 | 8.9ms | 22.1ms | 117% | 205 MB |
| SeaweedFS | 288 | 237 | 7.9ms | 57.0ms | 457% | 1081 MB |
| MinIO | 250 | 222 | 19.6ms | 33.7ms | 99% | 556 MB |
Garage leads concurrent writes at 504 ops/s. RustFS with single volume now matches Garage (503 vs 468 in Round 2 with 4 volumes). SeaweedFS p95 is terrible (57ms).
| Backend | Total ops/s | MB/s | Read p50 | Write p50 | CPU | Mem |
|---|---|---|---|---|---|---|
| HS5 | 651 | 524 | 5.4ms | 67.5ms | 34% | 227 MB |
| MinIO | 605 | 484 | 6.1ms | 32.3ms | 90% | 596 MB |
| Garage | 596 | 496 | 5.5ms | 15.6ms | 226% | 45 MB |
| RustFS | 586 | 480 | 6.3ms | 24.1ms | 104% | 949 MB |
| SeaweedFS | 564 | 464 | 5.2ms | 24.2ms | 296% | 1090 MB |
| Garage-zstd | 540 | 431 | 5.4ms | 12.3ms | 307% | 17 MB |
HS5 leads in total throughput but its write p50 spikes to 67ms under mixed contention (reads starve writes via WAL lock). Garage has the best write latency under mixed load (15.6ms).
| Backend | p50 | p95 |
|---|---|---|
| HS5 | 61ms | 72ms |
| Garage | 71ms | 76ms |
| Garage-zstd | 71ms | 84ms |
| MinIO | 101ms | 130ms |
| SeaweedFS | 151ms | 165ms |
| RustFS | 457ms | 474ms |
RustFS LIST improved from 1121ms (Round 2, 4 volumes) to 457ms (single volume) — still 7.5× slower than HS5.
| Metric | Garage (none) | Garage-zstd | Impact |
|---|---|---|---|
| Disk used | 608 MB | 120 MB | 5.1× smaller |
| Bulk upload | 338 MB/s | 309 MB/s | -9% |
| Seq read | 264 ops/s | 193 ops/s | -27% |
| Seq write | 193 ops/s | 163 ops/s | -16% |
| Conc write | 504 ops/s | 452 ops/s | -10% |
| Conc write CPU | 330% | 492% | +49% more CPU |
| Idle memory | 53 MB | 13 MB | 4× less (compressed LMDB) |
Verdict: Garage-zstd saves 5× disk space at the cost of 10–27% performance and 49% more CPU under write load. For write-once filing data where reads are infrequent, the disk savings are worth it. For read-heavy workloads, uncompressed is better.
| Backend | fsync Policy | Durability Guarantee | Impact on Write Performance |
|---|---|---|---|
| MinIO | O_DSYNC per write | ✅ Strongest — data survives power loss | Slowest: 74 ops/s seq write, p50=13.2ms |
| RustFS | No fsync per write | ❌ Data loss on crash | Fast: 180 ops/s, p50=3.4ms |
| SeaweedFS | No fsync by default | ❌ Data loss on crash (configurable per path via fs.configure -fsync) |
Fast: 74 ops/s (slow due to 3-container overhead, not fsync) |
| Garage | No fsync (our config: metadata_fsync=false, data_fsync=false) |
❌ Data loss on crash (can enable fsync in config) | Fast: 193 ops/s, p50=3.0ms |
| HS5 | No fsync (--manual-commit) |
❌ Data loss on crash (can trigger explicit commit for fsync) | Fastest: 212 ops/s, p50=2.4ms |
MinIO's slow writes are the cost of real durability. At 74 ops/s sequential writes with p50=13.2ms, MinIO is 2.4–2.9× slower than every other backend. But it's the only one that guarantees your data survives a power failure without any special configuration.
Evidence from raw disk benchmark: fdatasync costs 2.3ms per call (395 IOPS). MinIO's 13.2ms p50 = ~2.3ms fdatasync + ~11ms S3/metadata overhead. All other backends skip fsync entirely, which is why their p50 is 2.4–5.1ms (pure S3 overhead, no disk wait).
RustFS does NOT fsync despite being marketed as a MinIO replacement. Its 180 ops/s at p50=3.4ms proves it — if it used O_DSYNC like MinIO, it would be ~74 ops/s. This is a significant behavioral difference from MinIO.
For this project (write-once filing data re-fetchable from EDINET/TDnet): durability is nice-to-have, not critical. The data can always be re-scraped from source. But if you care about crash safety, only MinIO provides it out of the box.
| Metric | Round 2 (4 vol) | Round 3 (1 vol) | Change |
|---|---|---|---|
| Disk ratio | 2.07× | 1.02× | Fixed ✅ |
| Idle memory | 328 MB | 345 MB | ~same |
| Seq write | 156 ops/s | 180 ops/s | +15% |
| Conc write | 468 ops/s | 503 ops/s | +7% |
| List p50 | 1121ms | 457ms | 2.5× faster |
Single volume eliminates erasure coding overhead — RustFS is now competitive on all metrics.
| Category | 🥇 Winner | 🥈 Runner-up | 🥉 Third |
|---|---|---|---|
| Seq read ops/s | HS5 (616) | MinIO (447) | RustFS (399) |
| Seq write ops/s | HS5 (212) | Garage (193) | RustFS (180) |
| Conc read ops/s | HS5 (632) | Garage (616) | MinIO (597) |
| Conc write ops/s | Garage (504) | RustFS (503) | Garage-zstd (452) |
| Mixed throughput | HS5 (651) | MinIO (605) | Garage (596) |
| List p50 | HS5 (61ms) | Garage (71ms) | MinIO (101ms) |
| Disk efficiency | Garage-zstd (0.20×) | SeaweedFS (~0.36×) | Garage/MinIO/RustFS (1.0×) |
| Idle memory | Garage-zstd (13 MB) | Garage (53 MB) | HS5 (69 MB) |
| CPU efficiency | HS5 (34% mixed) | MinIO (90%) | RustFS (104%) |
| Bulk upload | RustFS (370 MB/s) | HS5 (358) | Garage (338) |
| Write latency p50 | HS5 (2.4ms seq) | Garage (3.0ms) | RustFS (3.4ms) |
| Rank | Backend | Strengths | Weaknesses |
|---|---|---|---|
| 1 | HS5 | Fastest reads + writes, lowest CPU, best LIST | 1.72× disk (pre-allocation), write starvation under mixed load |
| 2 | Garage | Best concurrent writes, low memory (53 MB), simple | High CPU under load (330%), slower sequential reads |
| 3 | Garage-zstd | Best disk efficiency (0.20×), lowest memory (13 MB) | 10–27% slower than uncompressed, highest CPU |
| 4 | RustFS | Fast bulk upload, good concurrent writes | 345 MB idle, 457ms LIST, 949 MB under mixed load |
| 5 | MinIO | Most compatible, decent mixed throughput | Slowest writes (74 ops/s seq), 424 MB idle |
| 6 | SeaweedFS | Good compression (~0.36×) | 804 MB idle (3 containers), 57ms write p95, highest resource usage |
Generated by scripts/storage-bench/single_backend_bench.py — each backend tested in isolation with fresh volumes
JSON results: results/comprehensive/single_*.json