Reproducing PyLate / LateOn SciFact (BEIR) on Firn 0.7.1 - ndcg@10 0.7575, latency, and cache hit/miss metrics

0-README.md

Firn 0.7.1 — PyLate / LateOn SciFact reproduction

Results and infra recipe for reproducing an earlier PyLate-on-Firn BEIR SciFact run on local hardware.

Files

• firn-beir-scifact.md — the writeup. Quality, latency, cache metrics, follow-ups.
• docker-compose.override.yml — drops into a firnflow checkout to pin Firn 0.7.1 and inject the cache / body-limit env vars used in the run.
• Dockerfile.bench — image recipe for running the bench scripts (pylate + accelerate + beir + ranx, with the torch 2.9 / torchvision ABI workaround).

Follow-ups

• nprobes sweep (100, 20, 8, 1): no measurable effect on quality or latency at this scale — MaxSim scoring dominates the IVF probe step. https://gist.github.com/gordonmurray/da9449ce797f4382ce49a895c10e8607

What's not here

The three bench scripts (01_encode.py, 02_upsert_and_index.py, 03_search_and_eval.py) are a compacted variant of PyLate's public BEIR boilerplate. Start from the upstream version: https://github.com/lightonai/pylate/blob/main/examples/evaluation/beir_dataset.py

Links

• Firn: https://github.com/gordonmurray/firnflow
• PyLate: https://github.com/lightonai/pylate
• Model: https://huggingface.co/lightonai/LateOn

1-firn-beir-scifact.md

Reproducing the SciFact BEIR bench on Firn 0.7.1

End-to-end reproduction of an earlier PyLate-on-Firn SciFact run, using the same bench scripts and parameters on local hardware.

Headline: quality numbers reproduce.

Quality

	Reference run	Local run
map	0.7118	0.7105
ndcg@10	0.7553	0.7575
ndcg@100	0.7753	0.7742
recall@10	0.8902	0.9036
recall@100	0.9767	0.9767

Same model (lightonai/LateOn), same num_sub_vectors=64, nprobes=100, document_length=300, k=100. Within normal run-to-run variance on every metric, and recall@100 lands exactly the same.

Latency / QPS

Two runs back-to-back, the second hitting an already-populated cache:

	Reference cold	Local cold	Reference warm	Local warm
QPS	2	3.1	13	210
p50	2 419 ms	9 936 ms	1 762 ms	149 ms
p95	56 115 ms	14 257 ms	6 803 ms	191 ms
p99	63 892 ms	17 478 ms	6 884 ms	240 ms

• The local cold distribution is more uniform but slower at the median than the reference. p50 is ~4× higher, p95/p99 are ~4× lower. Same orders of magnitude on both, but the shape differs — hardware and co-location (CPU core count, NIC vs loopback, where MinIO sits relative to firnflow) explain most of that, not anything in the bench config.
• Local warm is fast because the cold run on this host reached every query before the warm pass started; the foyer was fully populated for the warm run (cache_hits_total went from 0 → 300, zero new S3 query ops). 300 queries served in 1.4 s out of RAM / NVMe is what foyer does once the result set is in tier.

Cache effectiveness from /metrics

	After cold	After warm	Delta
cache_misses_total{ns=beir-scifact}	300	300	0
cache_hits_total{ns=beir-scifact}	0	300	+300
s3_requests_total{ns=beir-scifact, op=query}	300	300	0
query_duration_seconds_count	300	600	+300
query_duration_seconds_sum	2957.74	2957.76	+0.014 s

The 14 ms total added to the duration histogram across 300 cache-hit queries is the firnflow-side cost of a hit. Zero S3 query operations were issued during the warm pass.

Setup details

• Hardware. Local workstation. NVIDIA RTX 3080 (10 GB VRAM), 30 GB RAM, 1.6 TB free NVMe. MinIO, firnflow, the encoder, and the search driver are all co-located on the same box.
• Firn. ghcr.io/gordonmurray/firnflow:0.7.1, brought up via the repo's docker-compose.yml with a small override pinning the image and injecting FIRNFLOW_CACHE_NVME_BYTES=536870912, FIRNFLOW_CACHE_NVME_PATH=/tmp/firn-cache, and FIRNFLOW_MAX_BODY_BYTES=33554432.
• Bench container. PyTorch CUDA runtime image (pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime) with pylate, accelerate, beir, ranx, requests, srsly, tqdm, numpy pip-installed. torchvision / torchaudio get removed after install to dodge a torch 2.9 / torchvision 0.20 ABI mismatch that PyLate's transitive pulls created.
• GPU passthrough. A one-time host install of nvidia-container-toolkit and nvidia-ctk runtime configure --runtime=docker to switch Docker's CDI plumbing on.
• One env tweak. Encoder BATCH_SIZE dropped from 3000 to 64 to fit a 10 GB card. SciFact is small (5 183 docs), so the wall-clock cost is minor: 18.2 s document encode, 0.2 s query encode. Everything else matches the reference config verbatim (UPSERT_BATCH_SIZE=20, INDEX_NPROBES=100, INDEX_NUM_SUB_VECTORS=64, QUERY_CONCURRENCY=32, DOCUMENT_LENGTH=300, MODEL_NAME=lightonai/LateOn).

Wall-clock for the run:

• Encode (GPU): 18.4 s (5 183 docs + 300 queries)
• Upsert: 59.4 s, 87 docs/s, batches of 20 over HTTP
• Index build (IVF_PQ, async, observed via metric): 17.2 s
• Cold search (300 queries, concurrency 32): 96.7 s
• Warm search (same 300 queries): 1.4 s

Follow-ups

Quality matched, so the next sweeps are the ones the bench was built for:

1. INDEX_NPROBES=20 re-run. The reference config is 100; the script default is 20. If recall holds within ±0.005 at 20, the recommended default can drop substantially and cold latency should follow. Search-only re-run, same embeddings, same namespace.
2. num_partitions=5859 re-run. Firn currently inherits Lance's sqrt(row_count) ≈ 72 partitions for SciFact. PLAID derives total_tokens / 256 ≈ 5 859, two orders of magnitude higher. This is the parameter most likely to explain any remaining gap and needs a re-index pass (delete + upsert + index with explicit num_partitions).
3. Hardware comparison. Cold p50 diverges 4× in opposite directions depending on percentile. Worth comparing host specs (CPU core count, RAM, whether MinIO is co-located) before drawing any conclusions about Firn behaviour from those numbers.
4. Larger BEIR datasets. NFCorpus and FIQA next, then the memory pressure story on Quora and HotpotQA. Encoder BATCH_SIZE will likely need to come down further on the 10 GB card for the bigger document sets; nothing else changes.

2-docker-compose.override.yml

services:
  firnflow:
    image: ghcr.io/gordonmurray/firnflow:0.7.1
    build: !reset null
    environment:
      FIRNFLOW_S3_ENDPOINT: http://minio:9000
      FIRNFLOW_S3_ACCESS_KEY: minioadmin
      FIRNFLOW_S3_SECRET_KEY: minioadmin
      FIRNFLOW_S3_BUCKET: firnflow
      FIRNFLOW_BIND: "0.0.0.0:3000"
      FIRNFLOW_CACHE_NVME_BYTES: "536870912"
      FIRNFLOW_CACHE_NVME_PATH: /tmp/firn-cache
      FIRNFLOW_MAX_BODY_BYTES: "33554432"
      RUST_LOG: info

3-Dockerfile.bench

FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime

ENV PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    HF_HUB_DISABLE_TELEMETRY=1

RUN pip install --upgrade pip && \
    pip install \
      pylate \
      accelerate \
      beir \
      ranx \
      requests \
      srsly \
      tqdm \
      numpy && \
    pip uninstall -y torchvision torchaudio

WORKDIR /work