Skip to content

Instantly share code, notes, and snippets.

@benwtrent

benwtrent/dense_vector_hnsw.txt

Created October 10, 2024 16:57
Show Gist options
  • Save benwtrent/d620e4db7d6eb47f2bd64c3248dc3295 to your computer and use it in GitHub Desktop.
Save benwtrent/d620e4db7d6eb47f2bd64c3248dc3295 to your computer and use it in GitHub Desktop.
Download ZIP
bbq local runs
Raw
dense_vector_hnsw.txt
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Baseline | Contender | Diff | Unit | Diff % |
|--------------------------------------------------------------:|---------------------------------------------:|---------------:|-----------------:|--------------:|-------:|---------:|
| Cumulative indexing time of primary shards | | 11.9009 | 11.8778 | -0.02315 | min | -0.19% |
| Min cumulative indexing time across primary shard | | 5.925 | 5.90165 | -0.02335 | min | -0.39% |
| Median cumulative indexing time across primary shard | | 5.95047 | 5.9389 | -0.01158 | min | -0.19% |
| Max cumulative indexing time across primary shard | | 5.97595 | 5.97615 | 0.0002 | min | 0.00% |
| Cumulative indexing throttle time of primary shards | | 0 | 0 | 0 | min | 0.00% |
| Min cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Median cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Max cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% |
| Cumulative merge time of primary shards | | 24.3824 | 19.5845 | -4.79792 | min | -19.68% |
| Cumulative merge count of primary shards | | 16 | 16 | 0 | | 0.00% |
| Min cumulative merge time across primary shard | | 12.1129 | 9.66255 | -2.45037 | min | -20.23% |
| Median cumulative merge time across primary shard | | 12.1912 | 9.79224 | -2.39896 | min | -19.68% |
| Max cumulative merge time across primary shard | | 12.2695 | 9.92193 | -2.34755 | min | -19.13% |
| Cumulative merge throttle time of primary shards | | 0.662167 | 0.555717 | -0.10645 | min | -16.08% |
| Min cumulative merge throttle time across primary shard | | 0.33045 | 0.274483 | -0.05597 | min | -16.94% |
| Median cumulative merge throttle time across primary shard | | 0.331083 | 0.277858 | -0.05322 | min | -16.08% |
| Max cumulative merge throttle time across primary shard | | 0.331717 | 0.281233 | -0.05048 | min | -15.22% |
| Cumulative refresh time of primary shards | | 0.133333 | 0.148983 | 0.01565 | min | +11.74% |
| Cumulative refresh count of primary shards | | 185 | 190 | 5 | | +2.70% |
| Min cumulative refresh time across primary shard | | 0.0639 | 0.0738 | 0.0099 | min | +15.49% |
| Median cumulative refresh time across primary shard | | 0.0666667 | 0.0744917 | 0.00783 | min | +11.74% |
| Max cumulative refresh time across primary shard | | 0.0694333 | 0.0751833 | 0.00575 | min | +8.28% |
| Cumulative flush time of primary shards | | 0.30605 | 0.32975 | 0.0237 | min | +7.74% |
| Cumulative flush count of primary shards | | 16 | 16 | 0 | | 0.00% |
| Min cumulative flush time across primary shard | | 0.148017 | 0.163817 | 0.0158 | min | +10.67% |
| Median cumulative flush time across primary shard | | 0.153025 | 0.164875 | 0.01185 | min | +7.74% |
| Max cumulative flush time across primary shard | | 0.158033 | 0.165933 | 0.0079 | min | +5.00% |
| Total Young Gen GC time | | 4.834 | 4.859 | 0.025 | s | +0.52% |
| Total Young Gen GC count | | 121 | 207 | 86 | | +71.07% |
| Total Old Gen GC time | | 0 | 0 | 0 | s | 0.00% |
| Total Old Gen GC count | | 0 | 0 | 0 | | 0.00% |
| Dataset size | | 1.16234 | 1.25906 | 0.09672 | GB | +8.32% |
| Store size | | 1.16234 | 1.25906 | 0.09672 | GB | +8.32% |
| Translog size | | 1.02445e-07 | 1.02445e-07 | 0 | GB | 0.00% |
| Heap used for segments | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for doc values | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for terms | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for norms | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for points | | 0 | 0 | 0 | MB | 0.00% |
| Heap used for stored fields | | 0 | 0 | 0 | MB | 0.00% |
| Segment count | | 2 | 2 | 0 | | 0.00% |
| Total Ingest Pipeline count | | 0 | 0 | 0 | | 0.00% |
| Total Ingest Pipeline time | | 0 | 0 | 0 | ms | 0.00% |
| Total Ingest Pipeline failed | | 0 | 0 | 0 | | 0.00% |
| Min Throughput | index-append | 5593.82 | 5650.32 | 56.5002 | docs/s | +1.01% |
| Mean Throughput | index-append | 6132.83 | 6220.71 | 87.8793 | docs/s | +1.43% |
| Median Throughput | index-append | 5921.85 | 6012.53 | 90.6805 | docs/s | +1.53% |
| Max Throughput | index-append | 8012.63 | 8000.7 | -11.931 | docs/s | -0.15% |
| 50th percentile latency | index-append | 944.559 | 928.757 | -15.8017 | ms | -1.67% |
| 90th percentile latency | index-append | 1147.3 | 1118.21 | -29.0902 | ms | -2.54% |
| 99th percentile latency | index-append | 1341.78 | 1479.44 | 137.665 | ms | +10.26% |
| 100th percentile latency | index-append | 1617.91 | 1629.35 | 11.436 | ms | +0.71% |
| 50th percentile service time | index-append | 944.559 | 928.757 | -15.8017 | ms | -1.67% |
| 90th percentile service time | index-append | 1147.3 | 1118.21 | -29.0902 | ms | -2.54% |
| 99th percentile service time | index-append | 1341.78 | 1479.44 | 137.665 | ms | +10.26% |
| 100th percentile service time | index-append | 1617.91 | 1629.35 | 11.436 | ms | +0.71% |
| error rate | index-append | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | refresh-after-index | 7.26867 | 7.31951 | 0.05084 | ops/s | +0.70% |
| Mean Throughput | refresh-after-index | 7.26867 | 7.31951 | 0.05084 | ops/s | +0.70% |
| Median Throughput | refresh-after-index | 7.26867 | 7.31951 | 0.05084 | ops/s | +0.70% |
| Max Throughput | refresh-after-index | 7.26867 | 7.31951 | 0.05084 | ops/s | +0.70% |
| 100th percentile latency | refresh-after-index | 132.508 | 133.41 | 0.90225 | ms | +0.68% |
| 100th percentile service time | refresh-after-index | 132.508 | 133.41 | 0.90225 | ms | +0.68% |
| error rate | refresh-after-index | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | index-update-concurrent-with-searches | 7934.92 | 7164.75 | -770.171 | docs/s | -9.71% |
| Mean Throughput | index-update-concurrent-with-searches | 8128.51 | 7554.52 | -573.985 | docs/s | -7.06% |
| Median Throughput | index-update-concurrent-with-searches | 8166.55 | 7564.17 | -602.382 | docs/s | -7.38% |
| Max Throughput | index-update-concurrent-with-searches | 8473.63 | 7680.51 | -793.117 | docs/s | -9.36% |
| 50th percentile latency | index-update-concurrent-with-searches | 582.919 | 592.182 | 9.26356 | ms | +1.59% |
| 90th percentile latency | index-update-concurrent-with-searches | 698.081 | 754.378 | 56.2969 | ms | +8.06% |
| 100th percentile latency | index-update-concurrent-with-searches | 1267.04 | 945.474 | -321.569 | ms | -25.38% |
| 50th percentile service time | index-update-concurrent-with-searches | 582.919 | 592.182 | 9.26356 | ms | +1.59% |
| 90th percentile service time | index-update-concurrent-with-searches | 698.081 | 754.378 | 56.2969 | ms | +8.06% |
| 100th percentile service time | index-update-concurrent-with-searches | 1267.04 | 945.474 | -321.569 | ms | -25.38% |
| error rate | index-update-concurrent-with-searches | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | knn-search-100-1000-concurrent-with-indexing | 57.0297 | 55.4492 | -1.58059 | ops/s | -2.77% |
| Mean Throughput | knn-search-100-1000-concurrent-with-indexing | 59.8951 | 57.2233 | -2.67183 | ops/s | -4.46% |
| Median Throughput | knn-search-100-1000-concurrent-with-indexing | 59.86 | 57.1201 | -2.73994 | ops/s | -4.58% |
| Max Throughput | knn-search-100-1000-concurrent-with-indexing | 61.7387 | 59.8226 | -1.91608 | ops/s | -3.10% |
| 50th percentile latency | knn-search-100-1000-concurrent-with-indexing | 12.6718 | 12.8121 | 0.14037 | ms | +1.11% |
| 90th percentile latency | knn-search-100-1000-concurrent-with-indexing | 27.4597 | 27.9489 | 0.48916 | ms | +1.78% |
| 99th percentile latency | knn-search-100-1000-concurrent-with-indexing | 45.7359 | 52.382 | 6.64609 | ms | +14.53% |
| 99.9th percentile latency | knn-search-100-1000-concurrent-with-indexing | 74.6427 | 83.351 | 8.70833 | ms | +11.67% |
| 100th percentile latency | knn-search-100-1000-concurrent-with-indexing | 95.2265 | 162.27 | 67.0437 | ms | +70.40% |
| 50th percentile service time | knn-search-100-1000-concurrent-with-indexing | 12.6718 | 12.8121 | 0.14037 | ms | +1.11% |
| 90th percentile service time | knn-search-100-1000-concurrent-with-indexing | 27.4597 | 27.9489 | 0.48916 | ms | +1.78% |
| 99th percentile service time | knn-search-100-1000-concurrent-with-indexing | 45.7359 | 52.382 | 6.64609 | ms | +14.53% |
| 99.9th percentile service time | knn-search-100-1000-concurrent-with-indexing | 74.6427 | 83.351 | 8.70833 | ms | +11.67% |
| 100th percentile service time | knn-search-100-1000-concurrent-with-indexing | 95.2265 | 162.27 | 67.0437 | ms | +70.40% |
| error rate | knn-search-100-1000-concurrent-with-indexing | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | refresh-after-update | 118.819 | 69.954 | -48.8648 | ops/s | -41.13% |
| Mean Throughput | refresh-after-update | 118.819 | 69.954 | -48.8648 | ops/s | -41.13% |
| Median Throughput | refresh-after-update | 118.819 | 69.954 | -48.8648 | ops/s | -41.13% |
| Max Throughput | refresh-after-update | 118.819 | 69.954 | -48.8648 | ops/s | -41.13% |
| 100th percentile latency | refresh-after-update | 4.20146 | 11.3358 | 7.13429 | ms | +169.81% |
| 100th percentile service time | refresh-after-update | 4.20146 | 11.3358 | 7.13429 | ms | +169.81% |
| error rate | refresh-after-update | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | knn-search-10-100_multiple_segments | 137.503 | 184.561 | 47.0579 | ops/s | +34.22% |
| Mean Throughput | knn-search-10-100_multiple_segments | 154.392 | 192.96 | 38.5681 | ops/s | +24.98% |
| Median Throughput | knn-search-10-100_multiple_segments | 154.355 | 196.301 | 41.9459 | ops/s | +27.17% |
| Max Throughput | knn-search-10-100_multiple_segments | 169.589 | 198.234 | 28.6443 | ops/s | +16.89% |
| 50th percentile latency | knn-search-10-100_multiple_segments | 4.06096 | 3.68971 | -0.37125 | ms | -9.14% |
| 90th percentile latency | knn-search-10-100_multiple_segments | 5.87106 | 4.70783 | -1.16323 | ms | -19.81% |
| 99th percentile latency | knn-search-10-100_multiple_segments | 14.1074 | 9.22121 | -4.88623 | ms | -34.64% |
| 99.9th percentile latency | knn-search-10-100_multiple_segments | 23.069 | 11.9003 | -11.1687 | ms | -48.41% |
| 100th percentile latency | knn-search-10-100_multiple_segments | 24.7347 | 27.77 | 3.03529 | ms | +12.27% |
| 50th percentile service time | knn-search-10-100_multiple_segments | 4.06096 | 3.68971 | -0.37125 | ms | -9.14% |
| 90th percentile service time | knn-search-10-100_multiple_segments | 5.87106 | 4.70783 | -1.16323 | ms | -19.81% |
| 99th percentile service time | knn-search-10-100_multiple_segments | 14.1074 | 9.22121 | -4.88623 | ms | -34.64% |
| 99.9th percentile service time | knn-search-10-100_multiple_segments | 23.069 | 11.9003 | -11.1687 | ms | -48.41% |
| 100th percentile service time | knn-search-10-100_multiple_segments | 24.7347 | 27.77 | 3.03529 | ms | +12.27% |
| error rate | knn-search-10-100_multiple_segments | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | knn-search-100-1000_multiple_segments | 90.8198 | 97.5608 | 6.74098 | ops/s | +7.42% |
| Mean Throughput | knn-search-100-1000_multiple_segments | 94.3694 | 99.7416 | 5.37222 | ops/s | +5.69% |
| Median Throughput | knn-search-100-1000_multiple_segments | 94.7367 | 100.039 | 5.30275 | ops/s | +5.60% |
| Max Throughput | knn-search-100-1000_multiple_segments | 96.3014 | 100.99 | 4.68836 | ops/s | +4.87% |
| 50th percentile latency | knn-search-100-1000_multiple_segments | 7.91746 | 7.78104 | -0.13642 | ms | -1.72% |
| 90th percentile latency | knn-search-100-1000_multiple_segments | 10.7076 | 9.90285 | -0.8047 | ms | -7.52% |
| 99th percentile latency | knn-search-100-1000_multiple_segments | 28.3487 | 24.1563 | -4.19236 | ms | -14.79% |
| 99.9th percentile latency | knn-search-100-1000_multiple_segments | 33.8058 | 32.2756 | -1.53017 | ms | -4.53% |
| 100th percentile latency | knn-search-100-1000_multiple_segments | 33.9 | 32.946 | -0.954 | ms | -2.81% |
| 50th percentile service time | knn-search-100-1000_multiple_segments | 7.91746 | 7.78104 | -0.13642 | ms | -1.72% |
| 90th percentile service time | knn-search-100-1000_multiple_segments | 10.7076 | 9.90285 | -0.8047 | ms | -7.52% |
| 99th percentile service time | knn-search-100-1000_multiple_segments | 28.3487 | 24.1563 | -4.19236 | ms | -14.79% |
| 99.9th percentile service time | knn-search-100-1000_multiple_segments | 33.8058 | 32.2756 | -1.53017 | ms | -4.53% |
| 100th percentile service time | knn-search-100-1000_multiple_segments | 33.9 | 32.946 | -0.954 | ms | -2.81% |
| error rate | knn-search-100-1000_multiple_segments | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | force-merge | 0.000795901 | 0.0010146 | 0.00022 | ops/s | +27.48% |
| Mean Throughput | force-merge | 0.000795901 | 0.0010146 | 0.00022 | ops/s | +27.48% |
| Median Throughput | force-merge | 0.000795901 | 0.0010146 | 0.00022 | ops/s | +27.48% |
| Max Throughput | force-merge | 0.000795901 | 0.0010146 | 0.00022 | ops/s | +27.48% |
| 100th percentile latency | force-merge | 1.25643e+06 | 985609 | -270825 | ms | -21.56% |
| 100th percentile service time | force-merge | 1.25643e+06 | 985609 | -270825 | ms | -21.56% |
| error rate | force-merge | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | knn-search-10-100 | 35.5747 | 126.385 | 90.8106 | ops/s | +255.27% |
| Mean Throughput | knn-search-10-100 | 79.2728 | 156.436 | 77.1634 | ops/s | +97.34% |
| Median Throughput | knn-search-10-100 | 84.6767 | 162.184 | 77.5071 | ops/s | +91.53% |
| Max Throughput | knn-search-10-100 | 108.38 | 174.992 | 66.6122 | ops/s | +61.46% |
| 50th percentile latency | knn-search-10-100 | 3.48069 | 3.01981 | -0.46088 | ms | -13.24% |
| 90th percentile latency | knn-search-10-100 | 6.70801 | 3.77468 | -2.93332 | ms | -43.73% |
| 99th percentile latency | knn-search-10-100 | 10.3818 | 5.44165 | -4.94011 | ms | -47.58% |
| 99.9th percentile latency | knn-search-10-100 | 14.4593 | 13.6925 | -0.76681 | ms | -5.30% |
| 100th percentile latency | knn-search-10-100 | 14.8064 | 155.384 | 140.577 | ms | +949.44% |
| 50th percentile service time | knn-search-10-100 | 3.48069 | 3.01981 | -0.46088 | ms | -13.24% |
| 90th percentile service time | knn-search-10-100 | 6.70801 | 3.77468 | -2.93332 | ms | -43.73% |
| 99th percentile service time | knn-search-10-100 | 10.3818 | 5.44165 | -4.94011 | ms | -47.58% |
| 99.9th percentile service time | knn-search-10-100 | 14.4593 | 13.6925 | -0.76681 | ms | -5.30% |
| 100th percentile service time | knn-search-10-100 | 14.8064 | 155.384 | 140.577 | ms | +949.44% |
| error rate | knn-search-10-100 | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | knn-search-100-1000 | 109.586 | 96.7311 | -12.8546 | ops/s | -11.73% |
| Mean Throughput | knn-search-100-1000 | 114.025 | 101.106 | -12.9184 | ops/s | -11.33% |
| Median Throughput | knn-search-100-1000 | 115.224 | 101.671 | -13.5528 | ops/s | -11.76% |
| Max Throughput | knn-search-100-1000 | 116.18 | 102.547 | -13.6325 | ops/s | -11.73% |
| 50th percentile latency | knn-search-100-1000 | 6.95029 | 7.9399 | 0.9896 | ms | +14.24% |
| 90th percentile latency | knn-search-100-1000 | 8.73053 | 9.99127 | 1.26074 | ms | +14.44% |
| 99th percentile latency | knn-search-100-1000 | 12.1731 | 11.8579 | -0.31519 | ms | -2.59% |
| 99.9th percentile latency | knn-search-100-1000 | 16.4346 | 13.1743 | -3.26026 | ms | -19.84% |
| 100th percentile latency | knn-search-100-1000 | 34.3662 | 13.5296 | -20.8366 | ms | -60.63% |
| 50th percentile service time | knn-search-100-1000 | 6.95029 | 7.9399 | 0.9896 | ms | +14.24% |
| 90th percentile service time | knn-search-100-1000 | 8.73053 | 9.99127 | 1.26074 | ms | +14.44% |
| 99th percentile service time | knn-search-100-1000 | 12.1731 | 11.8579 | -0.31519 | ms | -2.59% |
| 99.9th percentile service time | knn-search-100-1000 | 16.4346 | 13.1743 | -3.26026 | ms | -19.84% |
| 100th percentile service time | knn-search-100-1000 | 34.3662 | 13.5296 | -20.8366 | ms | -60.63% |
| error rate | knn-search-100-1000 | 0 | 0 | 0 | % | 0.00% |
| Min Throughput | script-score-query | 13.0891 | 10.0684 | -3.02069 | ops/s | -23.08% |
| Mean Throughput | script-score-query | 13.1945 | 11.5309 | -1.6636 | ops/s | -12.61% |
| Median Throughput | script-score-query | 13.1612 | 11.6916 | -1.46966 | ops/s | -11.17% |
| Max Throughput | script-score-query | 13.4172 | 11.8735 | -1.54368 | ops/s | -11.51% |
| 50th percentile latency | script-score-query | 71.4768 | 78.5782 | 7.1014 | ms | +9.94% |
| 90th percentile latency | script-score-query | 74.5234 | 81.1212 | 6.59778 | ms | +8.85% |
| 99th percentile latency | script-score-query | 130.94 | 125.356 | -5.58373 | ms | -4.26% |
| 99.9th percentile latency | script-score-query | 187.399 | 134.108 | -53.2912 | ms | -28.44% |
| 100th percentile latency | script-score-query | 191.725 | 228.396 | 36.6715 | ms | +19.13% |
| 50th percentile service time | script-score-query | 71.4768 | 78.5782 | 7.1014 | ms | +9.94% |
| 90th percentile service time | script-score-query | 74.5234 | 81.1212 | 6.59778 | ms | +8.85% |
| 99th percentile service time | script-score-query | 130.94 | 125.356 | -5.58373 | ms | -4.26% |
| 99.9th percentile service time | script-score-query | 187.399 | 134.108 | -53.2912 | ms | -28.44% |
| 100th percentile service time | script-score-query | 191.725 | 228.396 | 36.6715 | ms | +19.13% |
| error rate | script-score-query | 0 | 0 | 0 | % | 0.00% |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment