OpenCLIP multilabel zero-shot and linear-probe evaluation results (CLIP / SigLIP2 / PE-Core / BiomedCLIP)

__MULTILABEL_EVALUATION_REPORT.md

Multilabel zero-shot classification and linear probes

Consolidated from OpenCLIP multilabel experiments; report prepared 2026-09-29. Metrics below are percentages, except counts and labels/image. “mAP” means macro average precision, not classification accuracy.

Findings

CLIP-family models can score multiple labels independently, but these experiments do not support treating their raw scores as reliable, calibrated label decisions. The visual domain, checkpoint, prompts, and threshold calibration all matter.

• Plants: PE-Core-L improved original-prompt zero-shot mAP to 51.61%, versus 43.96% for SigLIP2-L and 30.32% for LAION-2B ViT-B/16. A supervised linear head on frozen PE-Core features reached 93.65% mAP / 89.47% micro F1.
• BigEarthNet RGB: PE-Core-L reached 33.39% zero-shot test mAP, versus 25.16% for SigLIP2-L. Its linear probe reached 69.03% mAP / 74.45% micro F1.
• NIH ChestX-ray14: BiomedCLIP loaded and worked in this checkout. It improved zero-shot ranking over the generic models, but even the selected prompt variant reached only 12.92% mAP / 64.33% macro AUROC. Linear probes improved this to approximately 19.4–19.6% mAP / 74.9–75.6% macro AUROC.
• Prompting: visual descriptions helped BigEarthNet development ranking; positive/negative comparisons helped plants and some NIH configurations but substantially hurt BigEarthNet. Six generic templates gave little improvement. There was no universally best prompt family.

The much larger probe gains suggest that frozen image embeddings contain useful task information which these text prototypes do not recover. The comparison does not isolate how much comes from label supervision, task-specific decision directions, or feature standardization. NIH remains the weakest transfer setting.

Data, checkpoints, and evaluation protocol

Dataset	Labels	Probe fitting images	Calibration images	Complete held-out evaluation
Plant Pathology 2021	6	15,744	1,024 from train	1,864 validation images
BigEarthNet v2 RGB	19	237,871	122,342 validation	119,825 test images
NIH ChestX-ray14	14	77,856	8,668 train-fold-0	25,596 test images

Plant has 16,768 training images; the probe excludes the same 1,024 calibration images used for the original zero-shot threshold fitting. BigEarthNet retains its official geographic splits and uses only RGB, without multispectral bands or geographic metadata. NIH uses patient-grouped fold 0 for calibration and the remaining training folds for probe fitting. Its test and calibration populations each contain 2,797 patients, with no shared patients.

The prompt development sweep used different plant/BigEarthNet sample sizes, specified below. NIH has no separate validation split in this protocol: its training fold is development data, and the official test split is held out.

Pinned dataset revisions:

Dataset	Hugging Face revision
timm/plant-pathology-2021	e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46
timm/bigearthnet-v2-rgb	edb997737d3888abe5257c71d8e4fabc3c8ec34d
timm/nih-chest-xray-14	c1bf579641b3256b4d49924436984b82bee5834d

Report name	OpenCLIP model	Pretrained tag
LAION-2B ViT-B/16	ViT-B-16	laion2b_s34b_b88k
SigLIP2-L	ViT-L-16-SigLIP2-256	webli
PE-Core-L	PE-Core-L-14-336	meta
BiomedCLIP	hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224	Hub checkpoint; omit --pretrained

BiomedCLIP's cached model repository revision was 9f341de24bfb00180f1b847274256e9b65a3a32e. It required no compatibility fixes. This is a practical comparison of checkpoints with different architectures, resolutions, and training data. LAION-2B was evaluated only on plants here; BiomedCLIP was evaluated only on NIH.

Images use deterministic native model evaluation preprocessing; grayscale images are converted to RGB. CUDA image inference uses bfloat16, followed by float32 normalization/storage. Text encoding and zero-shot scoring use float32. Class order comes from dataset metadata, and empty NIH label lists are retained. An empty list means none of the 14 findings is annotated, not that normality is proven.

What “zero-shot” and “probe” mean here

For normalized image features v, each class has an independent normalized text prototype. Each prompt embedding is normalized, the embeddings are averaged, and the average is normalized again. There is no softmax across labels, forced top-k, or requirement to predict at least one label.

Scoring option	Score for class c
cosine	v @ t_positive[c]
logit	exp(logit_scale) * (v @ t_positive[c]) + logit_bias
paired	exp(logit_scale) * (v @ (t_positive[c] - t_negative[c]))

Paired scoring retains the margin instead of applying a sigmoid, avoiding saturation before AP/AUROC calculation. It is rank-equivalent to a per-label positive/negative two-way softmax. Neither cosine nor a sigmoid applied to a CLIP/SigLIP score establishes calibrated class-presence probabilities.

• Original zero-shot ranking: frozen image/text encoders and untuned prompts; no target training labels fit the ranking function.
• Thresholded F1: frozen models with per-class thresholds selected to maximize calibration F1. These decisions use labeled calibration data.
• Prompt selection: choosing a family by labeled development mAP is another form of supervision, even though model weights remain frozen.
• Linear probe: supervised binary logistic heads fitted to training embeddings; regularization and thresholds selected on calibration data. The encoder stays frozen.

Original zero-shot results on complete held-out splits

All rows use the original two positive prompts per label and cosine scoring. F1 uses the separate calibration populations above. The ranking metrics do not depend on those fitted thresholds.

Dataset	Checkpoint	mAP	Macro AUROC	Macro F1	Micro F1
Plant	LAION-2B ViT-B/16	30.32	67.42	38.80	40.76
Plant	SigLIP2-L	43.96	74.98	48.28	47.28
Plant	PE-Core-L	51.61	80.67	53.61	51.33
BigEarthNet RGB	SigLIP2-L	25.16	65.60	31.65	42.85
BigEarthNet RGB	PE-Core-L	33.39	72.90	37.80	47.03
NIH	SigLIP2-L	9.70	56.86	14.41	23.75
NIH	PE-Core-L	8.63	54.44	13.37	21.78
NIH	BiomedCLIP	12.28	62.96	17.67	22.19

PE-Core-L improved on SigLIP2-L by 7.65 mAP points on plants and 8.24 points on BigEarthNet, but did not improve NIH zero-shot ranking. Better general-purpose performance did not guarantee better chest X-ray transfer.

For the earlier question specifically about full BigEarthNet validation, these are the ranking results across all 122,342 validation images. This split was used for threshold calibration, so it is separate from the held-out test table above.

Checkpoint	Full validation mAP	Full validation macro AUROC
SigLIP2-L	28.50	66.63
PE-Core-L	37.30	74.40

Per-class results are in the full comparison.

Prompt strategies and development comparison

All six tested families are available through --prompt-strategy in scripts/multilabel_zeroshot.py.

CLI option	Positive ensemble per label	Default score
baseline	Original two domain phrases	cosine
baseline-paired	Original two domain phrases	paired
domain-templates	Six domain templates around the class name/subject	cosine
visual-descriptions	Three class-specific descriptions or synonyms	cosine
baseline-plus-descriptions	Original two phrases plus three descriptions	cosine
descriptions-paired	Original two phrases plus three descriptions	paired

Every family supports all three datasets. All paired variants use the original two negative phrases per class; descriptions-paired uses five positive texts, not just the three descriptions. Each positive text gets equal weight in the embedding average. Negative prototypes are averaged separately.

Examples include “an RGB satellite image containing permanent crops,” “a satellite image showing orchards and vineyards in regular rows,” and “a close-up photo of an apple leaf with bright orange spots.” NIH descriptions include label names and candidate radiographic cues. These are experimental descriptions, not expert-validated definitions or diagnostic criteria.

Baseline remains the default. The named paired options automatically use paired scoring and reject a conflicting --score cosine or --score logit. Other strategies allow an explicit score override, including --prompt-strategy visual-descriptions --score paired. Such additional combinations were not part of the six-family sweep. Custom --prompts file.json remains available and is mutually exclusive with an explicit --prompt-strategy; custom paired prompts require --score paired. Resolved strategy, scoring mode, and exact texts are saved in each run's artifacts.

Development populations

Candidates were written before the sweep and were not revised after inspecting its results. Within each model/dataset run, all families reused identical cached image embeddings. No held-out evaluation images were used in this sweep.

Dataset	Development population	Fewest positives in any class
Plant	2,048 of 16,768 train images	136
BigEarthNet RGB	8,192 of 122,342 validation images	25
NIH	All 8,668 train-fold-0 images	14

Plant and BigEarthNet samples were selected by the lowest SHA256 hashes of 42:image_id, across the entire development split, without label stratification. They are not ordered prefixes and differ from the full-split populations above. Rare classes and prevalence differences make cross-split mAP comparisons misleading.

Development mAP (%)

Prompt strategy	Plant PE-Core	BigEarthNet PE-Core	NIH PE-Core	NIH SigLIP2	NIH BiomedCLIP
baseline	47.94	38.18	5.22	5.86	8.17
baseline-paired	55.46	25.74	6.74	7.27	9.55
domain-templates	48.00	38.11	4.96	5.71	8.42
visual-descriptions	53.36	40.62	5.34	5.83	8.66
baseline-plus-descriptions	51.78	40.09	5.30	5.91	8.76
descriptions-paired	58.24	31.64	5.86	7.17	9.89

Development macro AUROC (%)

Prompt strategy	Plant PE-Core	BigEarthNet PE-Core	NIH PE-Core	NIH SigLIP2	NIH BiomedCLIP
baseline	79.71	74.28	53.63	54.91	59.96
baseline-paired	80.69	63.64	61.18	62.39	67.83
domain-templates	79.79	73.59	52.34	53.98	61.04
visual-descriptions	81.48	76.71	52.36	54.34	60.08
baseline-plus-descriptions	80.99	76.06	53.04	54.82	60.74
descriptions-paired	82.03	71.39	56.41	61.18	67.16

Descriptions improved PE-Core BigEarthNet development mAP from 38.18 to 40.62; baseline paired scoring reduced it to 25.74. Plant's best development family was the description mixture with paired scoring, at 58.24 mAP. These are development results: the selected richer plant and BigEarthNet prompts were not subsequently evaluated on their complete held-out splits in this experiment.

All candidate prompts fit their model's context. Maximum token counts were 22/32 for plant PE-Core, 19/32 for BigEarthNet PE-Core, 21/32 for NIH PE-Core, 17/64 for NIH SigLIP2, and 18/256 for NIH BiomedCLIP. Custom prompts may exceed these limits; separate short descriptions are preferable to a long concatenation.

The visual-description approach is inspired by classification by description. Per-label positive/negative comparison also appears in CheXzero, whose model was trained on chest X-rays and reports; those results cannot be credited to prompting alone. Negation can be unreliable (NegBench). DualPrompt co-occurrence prompting was discussed as further work but was not implemented or evaluated here. This experiment is not a full reproduction of any of these methods.

BiomedCLIP: selected prompts on the full NIH test split

BiomedCLIP was trained on broad biomedical figure-caption data, rather than being a dedicated chest-radiograph checkpoint (official model card). The single winning family by calibration mAP was fixed before test inference. The higher-calibration-AUROC baseline-paired family was not substituted after seeing test results. Only the original baseline and this selected family were tested on the 25,596 held-out images.

BiomedCLIP prompts	Test mAP	Macro AUROC	Macro F1	Micro F1
baseline	12.28	62.96	17.67	22.19
descriptions-paired	12.92	64.33	18.44	22.00

The selected prompts improved test mAP by 0.63 percentage point using unrounded metrics. Thresholded behavior remained weak: 13.75% micro precision, 55.02% recall, and 4.25 predicted labels/image versus 1.06 annotated. Improved ranking did not translate into improved micro F1 or reliable medical decisions.

See the BiomedCLIP report, including its selection record, exact prompts, and raw scores.

Supervised linear probes on complete held-out splits

The probe uses normalized projected global image embeddings: 1,024 dimensions for PE-Core-L and 512 for BiomedCLIP. Feature standardization uses only training means and standard deviations. Independent binary logistic heads have unpenalized biases; there is no class weighting, nonlinear head, augmentation, or encoder fine-tuning.

Full-batch float64 L-BFGS minimizes mean_binary_cross_entropy + lambda * sum(weight**2) / (2 * num_classes). The regularization grid is [1, 0.1, 0.01, 0.001, 0.0001], selected by calibration macro AP, with a 1,500-iteration budget. Only the selected head is evaluated on the held-out split. Per-class thresholds maximize calibration F1.

Dataset / encoder	Original zero-shot mAP	Probe mAP	Probe macro AUROC	Probe macro F1	Probe micro F1
Plant / PE-Core-L	51.61	93.65	98.26	89.04	89.47
BigEarthNet / PE-Core-L	33.39	69.03	93.63	64.63	74.45
NIH / BiomedCLIP	12.28	19.39	74.91	24.85	33.68
NIH / PE-Core-L	8.63	19.61	75.56	25.15	32.74

The probe's mAP improvement over original prompts was approximately 42.04 points on plants, 35.64 on BigEarthNet, 7.11 on NIH BiomedCLIP, and 10.99 on NIH PE-Core. NIH PE-Core's weak text-based ranking did not prevent its visual features from supporting a probe comparable to BiomedCLIP. The small difference between the two NIH probes has not been tested for statistical significance.

Dataset / encoder	Selected lambda	Micro precision	Micro recall	Predicted labels/image	Annotated labels/image
Plant / PE-Core-L	0.01	89.54	89.41	1.08	1.08
BigEarthNet / PE-Core-L	0.001	70.64	78.69	3.34	3.00
NIH / BiomedCLIP	0.0001	24.68	52.99	2.28	1.06
NIH / PE-Core-L	0.001	24.01	51.48	2.28	1.06

Aggregate results hide important class differences. For example, PE-Core's BigEarthNet probe reached 99.34% AP for marine waters but 18.51% for beaches/dunes/ sands. Its NIH probe reached 41.04% AP for effusion but 4.12% for pneumonia and 3.51% for hernia. The BiomedCLIP NIH probe's calibrated hernia F1 was 0.00%, despite 78.92% AUROC. Ranking quality and a useful decision threshold are distinct issues.

All per-class AP/AUROC/F1 results, saved heads, and regularization diagnostics are in the linear-probe report.

Reproduction and script options

See the usage guide for dependencies, streaming, custom JSON format, and score definitions. Set PYTHONPATH=src to use this checkout. --revision pins the dataset, not the model weights.

Original full BigEarthNet PE-Core evaluation, including full validation calibration:

PYTHONPATH=src python scripts/multilabel_zeroshot.py \
  --dataset bigearthnet-v2-rgb \
  --revision edb997737d3888abe5257c71d8e4fabc3c8ec34d \
  --model PE-Core-L-14-336 --pretrained meta \
  --prompt-strategy baseline --calibrate per-class \
  --device cuda --amp --batch-size 128 --workers 4 --download-data \
  --limit 0 --calibration-limit 0 \
  --output results/multilabel/reproduce-bigearth-pecore

For original plant results, use its dataset/revision, --calibration-limit 1024, and --workers 0 to preserve the original ordered calibration prefix. For NIH, the preset automatically filters calibration to patient fold 0. SigLIP2 uses --model ViT-L-16-SigLIP2-256 --pretrained webli; the model table lists other choices.

The selected BiomedCLIP family is now directly expressible through the public script:

PYTHONPATH=src python scripts/multilabel_zeroshot.py \
  --dataset nih-chest-xray-14 \
  --revision c1bf579641b3256b4d49924436984b82bee5834d \
  --model hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 \
  --prompt-strategy descriptions-paired --calibrate per-class \
  --device cuda --amp --batch-size 128 --workers 4 --download-data \
  --output results/multilabel/reproduce-nih-biomedclip-paired

Each run saves metrics.json, exact prompts.json, and evaluation.npz; calibrated runs also save calibration.npz. Add --save-features to retain normalized image embeddings. Limits default to zero, meaning the complete split/fold. A positive limit takes a prefix; it does not recreate the hash samples used by the prompt sweep. Changing prompts requires fitting new thresholds on calibration data.

Probes can be refit from saved feature caches without rerunning the image encoders via scripts/linear_probe.py --features <manifest.json>. A feature manifest maps paths.train, paths.calibration, and paths.evaluation to NPZ files containing features, targets, image_ids, patient_ids, and classnames. All caches must use identical checkpoint/preprocessing and disjoint data partitions. The zero-shot script saves its evaluation/calibration features; building the probe's training cache requires a separate extraction pass with calibration images excluded.

Implementation update: scripts/linear_probe.py now includes dataset loading/extraction, optional embedding caches, single-label and multilabel fitting, and saved-head evaluation. It accepts HF classification datasets, CSV, and WebDataset inputs. Both task types use this single entry point. See the linear-probe guide and dataset configurations for the integrated workflow. The measurements above are unchanged; the new plant example uses a 10% hash holdout rather than the historical 1,024-image prefix.

A saved probe predicts logits with ((features - feature_mean) / feature_std) @ weight + bias; compare those logits to the saved per-class thresholds. The text encoder is not used by the probe.

Verification, limitations, and artifacts

Complete split counts, class order, label coverage, unique image IDs, identical comparison populations, and train/calibration/evaluation disjointness were checked. NIH patient IDs are also disjoint. All evaluated classes had positives and negatives, so no threshold fallback was needed. Empty multilabel targets were kept.

The logistic optimizer was compared against independent scikit-learn logistic regression. Saved heads reconstructed evaluation logits within approximately 6e-14; saved scores reproduced the reported metrics. The six new built-in prompt families match all 30 saved experimental prompt JSONs exactly. Automated tests cover scoring, thresholds, split guards, prompt options, saved configuration, custom JSON, and the probe optimizer.

Small numerical differences can occur when bfloat16 image inference uses different batch shapes. Re-extracting features for the probes retained the same displayed zero-shot mAP, but produced plant/BigEarthNet zero-shot micro F1 of 51.21/46.99, versus 51.33/47.03 in the original runs. The original table above preserves the original measurements; the linear-probe report records its own matched-cache comparison.

BigEarthNet's selected probe lambda 0.001 reached L-BFGS's default function-evaluation budget with maximum absolute gradient 1.36e-7. The unselected lambda 0.0001 reached 1,500 iterations with maximum gradient 5.89e-8. Other candidates stopped before their budgets. These are measured results from a fixed sweep, not a claim of exhaustive optimization or the best attainable linear probe.

No confidence intervals, external-domain validation, or pretraining-overlap audit were performed. Prompts were evaluated on limited development populations for plants/BigEarthNet, with few positives for some classes. NIH labels are report-mined and noisy; these metrics measure agreement with those labels, not clinical reliability. Probes use projected global embeddings; intermediate features or encoder fine-tuning could give different results and were not tested here.

This report is self-contained for the main measurements. Detailed reports are included alongside it in this gist; per-run metrics, prompt JSONs, feature caches and saved heads are not:

• Original full-split comparisons and per-class results
• Development prompt sweep
• BiomedCLIP original and selected-prompt evaluations
• Linear probes and per-class results
• Zero-shot evaluator, linear-probe implementation, and usage guide

_full-comparison.md

Full multilabel comparison

Frozen models with the same untuned positive prompt ensembles for each dataset. Macro AP and AUROC use no fitted thresholds. F1 uses per-class thresholds fitted only on separate calibration data.

Dataset	Evaluation images	Checkpoint	Macro AP	Macro AUROC	Macro F1	Micro F1
BigEarthNet v2 RGB	119,825	SigLIP2 ViT-L/16-256	25.16%	65.60%	31.65%	42.85%
BigEarthNet v2 RGB	119,825	PE-Core-L/14-336	33.39%	72.90%	37.80%	47.03%
NIH ChestX-ray14	25,596	SigLIP2 ViT-L/16-256	9.70%	56.86%	14.41%	23.75%
NIH ChestX-ray14	25,596	PE-Core-L/14-336	8.63%	54.44%	13.37%	21.78%
NIH ChestX-ray14	25,596	BiomedCLIP PubMedBERT ViT-B/16-224	12.28%	62.96%	17.67%	22.19%
Plant Pathology 2021	1,864	CLIP ViT-B/16 LAION-2B	30.32%	67.42%	38.80%	40.76%
Plant Pathology 2021	1,864	SigLIP2 ViT-L/16-256	43.96%	74.98%	48.28%	47.28%
Plant Pathology 2021	1,864	PE-Core-L/14-336	51.61%	80.67%	53.61%	51.33%

BigEarthNet: official test split; thresholds fitted on all 122,342 validation images. NIH: official test split of 2,797 patients; thresholds fitted on all train-fold-0 images, grouped by patient. Plant: all 1,864 validation images; thresholds fitted on the same first 1,024 training images as the earlier comparison.

Checked complete counts, unique image IDs, identical targets/prompts and evaluation/calibration populations across checkpoints, and no image or NIH patient overlap. All classes have positives and negatives; no fallback thresholds were needed.

BigEarthNet full validation ranking

This is the validation split used for threshold fitting. Only threshold-free ranking metrics are shown here.

Checkpoint	Validation images	Macro AP	Macro AUROC
SigLIP2 ViT-L/16-256	122,342	28.50%	66.63%
PE-Core-L/14-336	122,342	37.30%	74.40%

BigEarthNet v2 RGB: per-class ranking

Class	Test/evaluation positives	SigLIP2 ViT-L/16-256 AP / AUROC	PE-Core-L/14-336 AP / AUROC
agriculture_with_natural_vegetation	29,846	27.57% / 56.02%	26.25% / 56.15%
agro_forestry_areas	9,942	10.00% / 59.75%	9.98% / 60.67%
arable_land	50,052	68.01% / 74.11%	79.15% / 84.74%
beaches_dunes_sands	152	2.34% / 83.67%	1.24% / 75.78%
broad_leaved_forest	36,377	44.41% / 65.49%	38.54% / 62.61%
coastal_wetlands	117	1.43% / 78.09%	8.30% / 92.25%
complex_cultivation_patterns	22,078	19.35% / 55.88%	21.35% / 60.59%
coniferous_forest	39,043	43.69% / 66.89%	62.81% / 78.61%
industrial_commercial_units	2,018	13.02% / 76.26%	17.27% / 84.32%
inland_waters	16,846	45.44% / 78.85%	62.69% / 87.12%
inland_wetlands	4,519	8.76% / 69.89%	12.29% / 73.52%
marine_waters	11,854	15.52% / 32.73%	42.83% / 64.46%
mixed_forest	44,284	46.34% / 64.39%	61.17% / 75.34%
moors_heathland_sclerophyllous_vegetation	3,759	4.98% / 52.92%	10.49% / 71.94%
natural_grassland_sparse_vegetation	2,211	6.04% / 72.80%	2.43% / 61.17%
pastures	26,722	49.83% / 74.88%	61.54% / 80.50%
permanent_crops	5,710	4.81% / 51.37%	5.62% / 57.27%
transitional_woodland_shrub	40,523	42.08% / 62.31%	48.74% / 70.06%
urban_fabric	12,824	24.40% / 70.16%	61.83% / 87.92%

NIH ChestX-ray14: per-class ranking

Class	Test/evaluation positives	SigLIP2 ViT-L/16-256 AP / AUROC	PE-Core-L/14-336 AP / AUROC	BiomedCLIP PubMedBERT ViT-B/16-224 AP / AUROC
Atelectasis	3,279	13.25% / 50.70%	14.08% / 51.65%	13.78% / 53.25%
Cardiomegaly	1,069	9.88% / 61.45%	7.05% / 54.56%	22.21% / 81.07%
Consolidation	1,815	8.75% / 57.66%	5.94% / 43.92%	9.37% / 59.07%
Edema	925	5.03% / 60.38%	4.49% / 55.67%	7.13% / 68.29%
Effusion	4,658	19.37% / 50.94%	17.58% / 47.56%	35.88% / 70.62%
Emphysema	1,093	4.53% / 50.98%	4.79% / 51.50%	5.21% / 55.23%
Fibrosis	435	2.92% / 59.29%	3.56% / 65.09%	4.28% / 68.38%
Hernia	86	0.54% / 63.95%	0.45% / 62.15%	1.07% / 77.31%
Infiltration	6,112	32.05% / 60.09%	20.99% / 45.19%	22.78% / 46.81%
Mass	1,748	11.50% / 60.85%	7.46% / 53.51%	12.35% / 58.48%
Nodule	1,623	7.92% / 56.87%	9.42% / 58.96%	8.07% / 55.93%
Pleural_Thickening	1,143	5.16% / 53.22%	5.83% / 55.79%	6.55% / 58.72%
Pneumonia	555	3.09% / 59.15%	3.43% / 60.52%	3.99% / 63.20%
Pneumothorax	2,665	11.73% / 50.48%	15.69% / 56.10%	19.32% / 65.10%

Plant Pathology 2021: per-class ranking

Class	Test/evaluation positives	CLIP ViT-B/16 LAION-2B AP / AUROC	SigLIP2 ViT-L/16-256 AP / AUROC	PE-Core-L/14-336 AP / AUROC
complex	215	22.30% / 70.36%	16.92% / 67.54%	24.10% / 72.09%
frog_eye_leaf_spot	435	32.61% / 58.54%	47.28% / 78.19%	39.03% / 73.62%
healthy	463	50.71% / 82.11%	46.69% / 73.74%	71.80% / 87.53%
powdery_mildew	127	28.49% / 75.98%	64.10% / 82.65%	73.95% / 91.27%
rust	208	14.62% / 60.07%	52.27% / 85.32%	51.00% / 86.12%
scab	572	33.18% / 57.48%	36.53% / 62.45%	49.81% / 73.36%

Scope

Checkpoints differ in architecture, resolution and pretraining data; this is a practical model comparison. Models are frozen, but the reported F1 uses labeled calibration data. Prompts were not selected using evaluation labels. No confidence intervals or audit of pretraining overlap were performed. NIH results measure agreement with report-mined labels, not clinical validation.

Usage guide.

_linear-probe.md

Frozen-encoder multilabel linear probes

These are measured results on complete held-out evaluation splits. Only a linear classifier is trained; encoder weights remain frozen. This is supervised adaptation, using training labels, not zero-shot inference or full-model fine-tuning.

Dataset	Encoder	Training images	Calibration images	Evaluation images	Zero-shot mAP	Probe mAP	Probe macro AUROC	Probe macro F1	Probe micro F1
Plant Pathology	PE-Core-L	15,744	1,024	1,864	51.61%	93.65%	98.26%	89.04%	89.47%
BigEarthNet RGB	PE-Core-L	237,871	122,342	119,825	33.39%	69.03%	93.63%	64.63%	74.45%
NIH ChestX-ray14	BiomedCLIP	77,856	8,668	25,596	12.28%	19.39%	74.91%	24.85%	33.68%
NIH ChestX-ray14	PE-Core-L	77,856	8,668	25,596	8.63%	19.61%	75.56%	25.15%	32.74%

The zero-shot column uses the original two positive prompts for each label, with the same encoder and images. Small numerical differences from earlier runs can arise from bfloat16 image inference with different batch shapes. Prompt-selected NIH BiomedCLIP previously reached 12.92% test mAP; that is a separate baseline.

Protocol

• Features: deterministic native-resolution image preprocessing, normalized projected global image embeddings; bfloat16 encoder inference followed by float32 normalization and storage.
• Head: independent binary logistic regressions with bias, no class softmax, no class weighting, no nonlinear layers. Feature means and standard deviations are fitted on training data only.
• Optimization: full-batch float64 L-BFGS. Objective is mean binary cross-entropy plus lambda * sum(weight**2) / (2 * number_of_classes); bias is not penalized.
• Selection: fixed regularization grid [1, 0.1, 0.01, 0.001, 0.0001], chosen by calibration macro AP. Per-class F1 thresholds are fitted on calibration data. Only the selected head is scored on evaluation data. No learning-rate or test-dependent prompt search is involved.
• Plants: 15,744 training images after reserving the same 1,024 training calibration images as the earlier zero-shot baseline. All 1,864 official validation images remain held out.
• BigEarthNet: all 237,871 training images; full validation for selection/thresholds; full test for evaluation. This RGB-only probe does not use multispectral bands or geographic metadata.
• NIH: training folds other than fold 0 for fitting; all 8,668 fold-0 images for calibration; all 25,596 official test images for evaluation. Patient IDs are disjoint across all three sets.

Verification and limitations

The optimizer was checked against independent scikit-learn binary logistic regressions, including multilabel examples. Saved heads reproduce every evaluation logit within numerical tolerance. Metrics were recomputed from saved scores. All split counts, class vocabularies, label coverage, image IDs, patient IDs and training/calibration/evaluation disjointness were checked. All labels have positives and negatives; no fallback thresholds were needed.

This is one deterministic split and a fixed regularization sweep, not an exhaustive estimate of the best attainable linear probe. Frozen projected global embeddings can differ from probes on intermediate backbone features. No confidence intervals, external-domain validation or audit of encoder pretraining overlap was performed. NIH metrics measure agreement with the dataset labels, not clinical reliability.

BigEarthNet optimization budget note: the selected lambda 0.001 reached L-BFGS's default function-evaluation budget with maximum absolute gradient 1.36e-7. The unselected lambda 0.0001 reached 1,500 iterations with maximum absolute gradient 5.89e-8. The remaining candidates/runs stopped before these budgets. Full objective values, gradient residuals and calibration curves are preserved in each selection file.

Decision behavior and regularization

Run	Selected lambda	Zero-shot micro F1	Probe micro F1	Probe precision	Probe recall	Predicted labels/image	Annotated labels/image
plant	0.01	51.21%	89.47%	89.54%	89.41%	1.08	1.08
bigearth	0.001	46.99%	74.45%	70.64%	78.69%	3.34	3.00
nih	0.0001	22.19%	33.68%	24.68%	52.99%	2.28	1.06
nih-pecore	0.001	21.78%	32.74%	24.01%	51.48%	2.28	1.06

plant: per-class probe results

Label	Positives	AP	AUROC	F1
complex	215	76.75%	95.64%	74.13%
frog_eye_leaf_spot	435	93.61%	97.19%	86.11%
healthy	463	99.67%	99.86%	98.26%
powdery_mildew	127	99.24%	99.92%	96.80%
rust	208	97.05%	99.48%	90.18%
scab	572	95.59%	97.47%	88.77%

Metrics and full protocol, calibration selection, saved head.

bigearth: per-class probe results

Label	Positives	AP	AUROC	F1
agriculture_with_natural_vegetation	29,846	68.77%	86.72%	64.64%
agro_forestry_areas	9,942	84.40%	98.49%	76.72%
arable_land	50,052	92.09%	93.74%	83.33%
beaches_dunes_sands	152	18.51%	97.01%	16.28%
broad_leaved_forest	36,377	79.06%	89.24%	72.99%
coastal_wetlands	117	41.75%	98.70%	37.57%
complex_cultivation_patterns	22,078	66.80%	90.42%	65.63%
coniferous_forest	39,043	88.23%	93.48%	81.84%
industrial_commercial_units	2,018	54.49%	96.02%	53.06%
inland_waters	16,846	89.97%	96.96%	81.54%
inland_wetlands	4,519	50.91%	91.87%	51.02%
marine_waters	11,854	99.34%	99.87%	97.23%
mixed_forest	44,284	85.18%	90.96%	78.32%
moors_heathland_sclerophyllous_vegetation	3,759	59.45%	96.79%	58.68%
natural_grassland_sparse_vegetation	2,211	38.93%	92.15%	41.90%
pastures	26,722	82.81%	92.34%	71.92%
permanent_crops	5,710	52.03%	92.37%	49.72%
transitional_woodland_shrub	40,523	73.78%	85.23%	69.05%
urban_fabric	12,824	85.12%	96.69%	76.45%

Metrics and full protocol, calibration selection, saved head.

nih: per-class probe results

Label	Positives	AP	AUROC	F1
Atelectasis	3,279	25.92%	70.69%	32.76%
Cardiomegaly	1,069	25.95%	83.79%	29.81%
Consolidation	1,815	14.04%	71.36%	21.77%
Edema	925	13.72%	81.89%	22.02%
Effusion	4,658	42.70%	77.52%	46.29%
Emphysema	1,093	18.34%	78.64%	24.51%
Fibrosis	435	6.01%	78.54%	9.99%
Hernia	86	2.51%	78.92%	0.00%
Infiltration	6,112	37.76%	67.82%	45.73%
Mass	1,748	21.81%	74.29%	29.09%
Nodule	1,623	14.49%	66.68%	21.12%
Pleural_Thickening	1,143	10.82%	72.06%	17.25%
Pneumonia	555	4.05%	66.45%	7.44%
Pneumothorax	2,665	33.36%	80.10%	40.19%

Metrics and full protocol, calibration selection, saved head.

nih-pecore: per-class probe results

Label	Positives	AP	AUROC	F1
Atelectasis	3,279	26.68%	71.73%	33.53%
Cardiomegaly	1,069	19.98%	78.84%	24.03%
Consolidation	1,815	12.93%	70.10%	21.08%
Edema	925	13.20%	81.83%	21.10%
Effusion	4,658	41.04%	76.04%	45.09%
Emphysema	1,093	26.33%	82.36%	30.42%
Fibrosis	435	7.47%	78.14%	9.72%
Hernia	86	3.51%	87.13%	5.10%
Infiltration	6,112	37.92%	68.11%	45.29%
Mass	1,748	19.05%	72.48%	26.20%
Nodule	1,623	16.93%	69.76%	24.11%
Pleural_Thickening	1,143	10.04%	72.25%	17.83%
Pneumonia	555	4.12%	67.05%	7.95%
Pneumothorax	2,665	35.42%	81.96%	40.65%

Metrics and full protocol, calibration selection, saved head.

Reproduction

Run scripts/linear_probe.py with the dataset configurations in scripts/probe_configs/. head.npz contains the training-only standardization statistics, weight matrix, bias, thresholds and label order.

_prompt-comparison.md

Calibration-only prompt comparison

All numbers below are exploratory development results, not new held-out test results. Prompt candidates were written before running this sweep; no prompt was revised after viewing results. Choosing a winning family from these numbers uses labeled development data. No model weights were trained and no evaluation/test images were loaded.

PE-Core-L-14-336 uses meta; SigLIP2-L-16-256 uses webli; BiomedCLIP uses the Microsoft Hub checkpoint. Image inference uses bfloat16; normalized image embeddings, text embeddings and scoring use float32. Each family uses exactly the same cached image embeddings within a model/dataset run.

Macro average precision (%)

Prompt family	plant-pecore	bigearth-pecore	nih-pecore	nih-siglip2	nih-biomedclip
Original two prompts	47.94	38.18	5.22	5.86	8.17
Original positive/negative pairs	55.46	25.74	6.74	7.27	9.55
Six domain templates	48.00	38.11	4.96	5.71	8.42
Three visual descriptions / synonyms	53.36	40.62	5.34	5.83	8.66
Original prompts + descriptions	51.78	40.09	5.30	5.91	8.76
Original + descriptions, paired	58.24	31.64	5.86	7.17	9.89

Macro AUROC (%)

Prompt family	plant-pecore	bigearth-pecore	nih-pecore	nih-siglip2	nih-biomedclip
Original two prompts	79.71	74.28	53.63	54.91	59.96
Original positive/negative pairs	80.69	63.64	61.18	62.39	67.83
Six domain templates	79.79	73.59	52.34	53.98	61.04
Three visual descriptions / synonyms	81.48	76.71	52.36	54.34	60.08
Original prompts + descriptions	80.99	76.06	53.04	54.82	60.74
Original + descriptions, paired	82.03	71.39	56.41	61.18	67.16

Data and verification

• plant-pecore: 2,048 of 16,768 eligible images, split train, fold None, lowest SHA256(42:image_id), no label stratification. Fewest positives for any class: 136. Maximum prompt length 22 of 32 tokens. Metrics and per-class results.
• bigearth-pecore: 8,192 of 122,342 eligible images, split validation, fold None, lowest SHA256(42:image_id), no label stratification. Fewest positives for any class: 25. Maximum prompt length 19 of 32 tokens. Metrics and per-class results.
• nih-pecore: 8,668 of 8,668 eligible images, split train, fold 0, full fold. Fewest positives for any class: 14. Maximum prompt length 21 of 32 tokens. Metrics and per-class results.
• nih-siglip2: 8,668 of 8,668 eligible images, split train, fold 0, full fold. Fewest positives for any class: 14. Maximum prompt length 17 of 64 tokens. Metrics and per-class results.
• nih-biomedclip: 8,668 of 8,668 eligible images, split train, fold 0, full fold. Fewest positives for any class: 14. Maximum prompt length 18 of 256 tokens. Metrics and per-class results.

Every class has positives and negatives in each run. NIH uses all 8,668 images in training patient fold 0; its label prevalences differ from the test split. Plant and BigEarthNet were sampled deterministically by image ID hash across their whole development split, with no label stratification. Rare classes have few examples, so small mAP changes need confirmation. These samples are not spatially independent confidence intervals for satellite imagery.

Targets and patient IDs were checked against overlapping images in previous calibration runs. Original baseline macro AP reproduces within 0.01 percentage point on matching images; numerical score differences are consistent with bfloat16 inference with different partial batch shapes. Mean and maximum differences are recorded in summary.json. All current prompt comparisons use identical features. No prompt exceeded the tokenizer context length. The new feature-save path passed the evaluator tests (23 tests including the existing zero-shot evaluation tests).

Interpretation

• On PE-Core, descriptions helped BigEarthNet, while negated prompts hurt substantially. Plant benefited most from descriptions plus paired scoring. Six generic templates gave little benefit.
• NIH remains weak with these generic encoders. Compare paired scoring per encoder; richer text alone does not establish disease discrimination or calibrated predictions.
• Rankings are reported before choosing thresholds. Changing prompts requires fresh development threshold calibration before applying the frozen choice to held-out evaluation. No fitted F1 is reported on the same sample used to choose prompts.

Prompt formats

Each candidate family is a prompt JSON accepted by scripts/multilabel_zeroshot.py --prompts, and the six families are also built in as --prompt-strategy presets. Use --score paired for the paired families and --score cosine for the others. Descriptions use standard visual cues but are not expert-validated label definitions. This sweep is inspired by published ideas, not a complete reproduction of any paper.

Related methods

• Classification by Description / DCLIP: visual descriptors beyond label names.
• CheXzero: per-disease positive/negative comparison, with a model trained on chest X-ray/report pairs.
• NegBench: evidence that VLM negation understanding is unreliable.
• DualPrompt: discriminative and co-occurrence prompts; not tested here. Its data/prior variants use training labels.

full-comparison-summary.json

{
  "BigEarthNet v2 RGB": {
    "siglip2": {
      "model": "SigLIP2 ViT-L/16-256",
      "ranking": {
        "num_images": 119825,
        "num_classes": 19,
        "mean_labels": 2.9950093886918423,
        "macro_ap": 0.25157659876763727,
        "micro_ap": 0.281902255134278,
        "macro_auroc": 0.6560296311810878,
        "ap_classes": 19,
        "auroc_classes": 19,
        "per_class": {
          "agriculture_with_natural_vegetation": {
            "positives": 29846,
            "prevalence": 0.24907990819945755,
            "ap": 0.2756762385036491,
            "auroc": 0.560158961406183
          },
          "agro_forestry_areas": {
            "positives": 9942,
            "prevalence": 0.08297099937408721,
            "ap": 0.09997320568509971,
            "auroc": 0.597541867436393
          },
          "arable_land": {
            "positives": 50052,
            "prevalence": 0.4177091591904861,
            "ap": 0.6800744923815237,
            "auroc": 0.7411082314302546
          },
          "beaches_dunes_sands": {
            "positives": 152,
            "prevalence": 0.0012685165866889213,
            "ap": 0.02336228266585698,
            "auroc": 0.8367002659000162
          },
          "broad_leaved_forest": {
            "positives": 36377,
            "prevalence": 0.3035843939077822,
            "ap": 0.44408843265915443,
            "auroc": 0.6549020177012854
          },
          "coastal_wetlands": {
            "positives": 117,
            "prevalence": 0.0009764239515960776,
            "ap": 0.014270221187652027,
            "auroc": 0.7809139347340638
          },
          "complex_cultivation_patterns": {
            "positives": 22078,
            "prevalence": 0.18425203421656583,
            "ap": 0.19354840089358777,
            "auroc": 0.5587977342869388
          },
          "coniferous_forest": {
            "positives": 39043,
            "prevalence": 0.3258335071979971,
            "ap": 0.43693885445542974,
            "auroc": 0.6689463702233065
          },
          "industrial_commercial_units": {
            "positives": 2018,
            "prevalence": 0.01684122678906739,
            "ap": 0.1301684631699624,
            "auroc": 0.7626071149631838
          },
          "inland_waters": {
            "positives": 16846,
            "prevalence": 0.14058835802211558,
            "ap": 0.454357534211006,
            "auroc": 0.7884776450533502
          },
          "inland_wetlands": {
            "positives": 4519,
            "prevalence": 0.03771333194241602,
            "ap": 0.08758755555655481,
            "auroc": 0.6988563306272454
          },
          "marine_waters": {
            "positives": 11854,
            "prevalence": 0.09892760275401627,
            "ap": 0.1551729557529074,
            "auroc": 0.3272843767700423
          },
          "mixed_forest": {
            "positives": 44284,
            "prevalence": 0.36957229292718546,
            "ap": 0.46341282667001454,
            "auroc": 0.643881357049819
          },
          "moors_heathland_sclerophyllous_vegetation": {
            "positives": 3759,
            "prevalence": 0.03137074900897142,
            "ap": 0.049755485395735585,
            "auroc": 0.5291840149640668
          },
          "natural_grassland_sparse_vegetation": {
            "positives": 2211,
            "prevalence": 0.018451909034007927,
            "ap": 0.06035258304308987,
            "auroc": 0.7280130869420168
          },
          "pastures": {
            "positives": 26722,
            "prevalence": 0.2230085541414563,
            "ap": 0.4982534496200138,
            "auroc": 0.7488444614381004
          },
          "permanent_crops": {
            "positives": 5710,
            "prevalence": 0.04765282703943251,
            "ap": 0.04809584803368072,
            "auroc": 0.5136831558296072
          },
          "transitional_woodland_shrub": {
            "positives": 40523,
            "prevalence": 0.3381848529104945,
            "ap": 0.4208245166692249,
            "auroc": 0.6230824089354159
          },
          "urban_fabric": {
            "positives": 12824,
            "prevalence": 0.10702274149801794,
            "ap": 0.24404203003096459,
            "auroc": 0.7015796567493789
          }
        }
      },
      "decisions": {
        "micro_f1": 0.4285065128319497,
        "macro_f1": 0.3165326384759147,
        "micro_precision": 0.29477418908927927,
        "micro_recall": 0.7843467260370545,
        "exact_match": 0.000592530774045483,
        "mean_predicted_labels": 7.969238472772794,
        "per_class_f1": [
          0.4307179135002151,
          0.1836627801056659,
          0.6483535064605673,
          0.08071748878923767,
          0.5037950088044204,
          0.06382978723404255,
          0.3443682948683796,
          0.544465617079533,
          0.20382753055107217,
          0.49714285714285716,
          0.14434909864963746,
          0.21412866838734346,
          0.5638969320363081,
          0.08267203600615895,
          0.10653685674547983,
          0.48531031950055087,
          0.09917912167792493,
          0.533078080517664,
          0.28408823298532065
        ]
      },
      "calibration_images": 122342,
      "configuration": {
        "dataset": "bigearthnet-v2-rgb",
        "model": "ViT-L-16-SigLIP2-256",
        "pretrained": "webli",
        "device": "cuda:0",
        "amp": true,
        "batch_size": 128,
        "workers": 4,
        "download_data": true,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 0,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "edb997737d3888abe5257c71d8e4fabc3c8ec34d",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/bigearth-siglip2-full"
      },
      "evaluation_patients": null,
      "validation_ranking": {
        "num_images": 122342,
        "num_classes": 19,
        "mean_labels": 3.011042814405519,
        "macro_ap": 0.2849635952936094,
        "micro_ap": 0.2797418312646959,
        "macro_auroc": 0.6663422023895581,
        "ap_classes": 19,
        "auroc_classes": 19,
        "per_class": {
          "agriculture_with_natural_vegetation": {
            "positives": 32736,
            "prevalence": 0.26757777378169395,
            "ap": 0.29873960118430615,
            "auroc": 0.5675725486216197
          },
          "agro_forestry_areas": {
            "positives": 8157,
            "prevalence": 0.06667375063347011,
            "ap": 0.08267880265141511,
            "auroc": 0.6080517036458534
          },
          "arable_land": {
            "positives": 49737,
            "prevalence": 0.40654068104167007,
            "ap": 0.6550946393096486,
            "auroc": 0.734319979050137
          },
          "beaches_dunes_sands": {
            "positives": 426,
            "prevalence": 0.003482042144153275,
            "ap": 0.4649429413675562,
            "auroc": 0.9099130267788474
          },
          "broad_leaved_forest": {
            "positives": 35387,
            "prevalence": 0.28924653839237546,
            "ap": 0.4444139241513232,
            "auroc": 0.6648054554352276
          },
          "coastal_wetlands": {
            "positives": 610,
            "prevalence": 0.004986022788576286,
            "ap": 0.10165036212129963,
            "auroc": 0.789297889262788
          },
          "complex_cultivation_patterns": {
            "positives": 26486,
            "prevalence": 0.21649147471841232,
            "ap": 0.22818332564742994,
            "auroc": 0.563439166551118
          },
          "coniferous_forest": {
            "positives": 39554,
            "prevalence": 0.3233067957038466,
            "ap": 0.4548862485401354,
            "auroc": 0.6897705671926084
          },
          "industrial_commercial_units": {
            "positives": 2726,
            "prevalence": 0.022281800199440913,
            "ap": 0.15449860995138998,
            "auroc": 0.7855070117135902
          },
          "inland_waters": {
            "positives": 15900,
            "prevalence": 0.12996354481698844,
            "ap": 0.4662148243658854,
            "auroc": 0.8088895552885624
          },
          "inland_wetlands": {
            "positives": 5056,
            "prevalence": 0.0413267724902323,
            "ap": 0.08801814453276005,
            "auroc": 0.6820964675875072
          },
          "marine_waters": {
            "positives": 14064,
            "prevalence": 0.11495643360415883,
            "ap": 0.17587300365659386,
            "auroc": 0.3545153765438103
          },
          "mixed_forest": {
            "positives": 42705,
            "prevalence": 0.34906246423959064,
            "ap": 0.46474056178618584,
            "auroc": 0.6709477232100003
          },
          "moors_heathland_sclerophyllous_vegetation": {
            "positives": 3806,
            "prevalence": 0.031109512677575977,
            "ap": 0.04209433609157269,
            "auroc": 0.5150213915603256
          },
          "natural_grassland_sparse_vegetation": {
            "positives": 2687,
            "prevalence": 0.021963021693286035,
            "ap": 0.10377829944586615,
            "auroc": 0.7577770194880309
          },
          "pastures": {
            "positives": 25915,
            "prevalence": 0.2118242304359909,
            "ap": 0.43868714765343325,
            "auroc": 0.7235084782840976
          },
          "permanent_crops": {
            "positives": 8688,
            "prevalence": 0.07101404260188651,
            "ap": 0.06905895731065044,
            "auroc": 0.5006128338016378
          },
          "transitional_woodland_shrub": {
            "positives": 36429,
            "prevalence": 0.2977636461722058,
            "ap": 0.38222678289016393,
            "auroc": 0.630659165326957
          },
          "urban_fabric": {
            "positives": 17308,
            "prevalence": 0.14147226626996454,
            "ap": 0.29852779792096296,
            "auroc": 0.703796486058886
          }
        }
      }
    },
    "pecore": {
      "model": "PE-Core-L/14-336",
      "ranking": {
        "num_images": 119825,
        "num_classes": 19,
        "mean_labels": 2.9950093886918423,
        "macro_ap": 0.33394261705958916,
        "micro_ap": 0.3659722245235833,
        "macro_auroc": 0.7289642388872511,
        "ap_classes": 19,
        "auroc_classes": 19,
        "per_class": {
          "agriculture_with_natural_vegetation": {
            "positives": 29846,
            "prevalence": 0.24907990819945755,
            "ap": 0.26253956762843006,
            "auroc": 0.5615457603810863
          },
          "agro_forestry_areas": {
            "positives": 9942,
            "prevalence": 0.08297099937408721,
            "ap": 0.09981365018108034,
            "auroc": 0.606708338942022
          },
          "arable_land": {
            "positives": 50052,
            "prevalence": 0.4177091591904861,
            "ap": 0.7914851761495356,
            "auroc": 0.8474089293314706
          },
          "beaches_dunes_sands": {
            "positives": 152,
            "prevalence": 0.0012685165866889213,
            "ap": 0.012389277727951972,
            "auroc": 0.7577934410742959
          },
          "broad_leaved_forest": {
            "positives": 36377,
            "prevalence": 0.3035843939077822,
            "ap": 0.38536434091199084,
            "auroc": 0.6261292996999089
          },
          "coastal_wetlands": {
            "positives": 117,
            "prevalence": 0.0009764239515960776,
            "ap": 0.08296947218380507,
            "auroc": 0.9225399326395083
          },
          "complex_cultivation_patterns": {
            "positives": 22078,
            "prevalence": 0.18425203421656583,
            "ap": 0.21350703701962495,
            "auroc": 0.6058944249098416
          },
          "coniferous_forest": {
            "positives": 39043,
            "prevalence": 0.3258335071979971,
            "ap": 0.628056843548577,
            "auroc": 0.7861297918981341
          },
          "industrial_commercial_units": {
            "positives": 2018,
            "prevalence": 0.01684122678906739,
            "ap": 0.17268767770238883,
            "auroc": 0.8432216803881485
          },
          "inland_waters": {
            "positives": 16846,
            "prevalence": 0.14058835802211558,
            "ap": 0.6268933823349863,
            "auroc": 0.871171651425096
          },
          "inland_wetlands": {
            "positives": 4519,
            "prevalence": 0.03771333194241602,
            "ap": 0.12285338489054363,
            "auroc": 0.7351653857860428
          },
          "marine_waters": {
            "positives": 11854,
            "prevalence": 0.09892760275401627,
            "ap": 0.42825305818234777,
            "auroc": 0.6446209333619048
          },
          "mixed_forest": {
            "positives": 44284,
            "prevalence": 0.36957229292718546,
            "ap": 0.6116773611525426,
            "auroc": 0.7534195654617268
          },
          "moors_heathland_sclerophyllous_vegetation": {
            "positives": 3759,
            "prevalence": 0.03137074900897142,
            "ap": 0.1048883204515348,
            "auroc": 0.7193796835108361
          },
          "natural_grassland_sparse_vegetation": {
            "positives": 2211,
            "prevalence": 0.018451909034007927,
            "ap": 0.02432932007684252,
            "auroc": 0.6116669241994585
          },
          "pastures": {
            "positives": 26722,
            "prevalence": 0.2230085541414563,
            "ap": 0.6153782026406042,
            "auroc": 0.804960962581379
          },
          "permanent_crops": {
            "positives": 5710,
            "prevalence": 0.04765282703943251,
            "ap": 0.05620369197662769,
            "auroc": 0.5727437940327041
          },
          "transitional_woodland_shrub": {
            "positives": 40523,
            "prevalence": 0.3381848529104945,
            "ap": 0.48735707058368244,
            "auroc": 0.7006259498697863
          },
          "urban_fabric": {
            "positives": 12824,
            "prevalence": 0.10702274149801794,
            "ap": 0.6182628887890973,
            "auroc": 0.8791940893644205
          }
        }
      },
      "decisions": {
        "micro_f1": 0.4702909108485648,
        "macro_f1": 0.37796172808826894,
        "micro_precision": 0.3406715843994743,
        "micro_recall": 0.7591235994505081,
        "exact_match": 0.024752764448153556,
        "mean_predicted_labels": 6.673824327143751,
        "per_class_f1": [
          0.43201560830584845,
          0.18402306002463462,
          0.7352479668600429,
          0.04524886877828054,
          0.5066455448302274,
          0.1349206349206349,
          0.355970296851682,
          0.6225113032442294,
          0.25421715371822284,
          0.6375417944679322,
          0.18707198931017205,
          0.4640953276889307,
          0.6354109705700429,
          0.13091235149069716,
          0.04864540545530069,
          0.5524550856586857,
          0.10651477394090424,
          0.5747265882856684,
          0.573098109274974
        ]
      },
      "calibration_images": 122342,
      "configuration": {
        "dataset": "bigearthnet-v2-rgb",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "device": "cuda:1",
        "amp": true,
        "batch_size": 128,
        "workers": 4,
        "download_data": true,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 0,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "edb997737d3888abe5257c71d8e4fabc3c8ec34d",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/bigearth-pecore-full"
      },
      "evaluation_patients": null,
      "validation_ranking": {
        "num_images": 122342,
        "num_classes": 19,
        "mean_labels": 3.011042814405519,
        "macro_ap": 0.37297547812362397,
        "micro_ap": 0.36661544181117034,
        "macro_auroc": 0.744026281574781,
        "ap_classes": 19,
        "auroc_classes": 19,
        "per_class": {
          "agriculture_with_natural_vegetation": {
            "positives": 32736,
            "prevalence": 0.26757777378169395,
            "ap": 0.2926249791481824,
            "auroc": 0.5812819162237098
          },
          "agro_forestry_areas": {
            "positives": 8157,
            "prevalence": 0.06667375063347011,
            "ap": 0.08479734757584624,
            "auroc": 0.6270302840580296
          },
          "arable_land": {
            "positives": 49737,
            "prevalence": 0.40654068104167007,
            "ap": 0.773073005512408,
            "auroc": 0.8409400281095947
          },
          "beaches_dunes_sands": {
            "positives": 426,
            "prevalence": 0.003482042144153275,
            "ap": 0.55427802333522,
            "auroc": 0.9572787513052549
          },
          "broad_leaved_forest": {
            "positives": 35387,
            "prevalence": 0.28924653839237546,
            "ap": 0.38544502466645736,
            "auroc": 0.6450227838576855
          },
          "coastal_wetlands": {
            "positives": 610,
            "prevalence": 0.004986022788576286,
            "ap": 0.25243582404390835,
            "auroc": 0.8893437640223376
          },
          "complex_cultivation_patterns": {
            "positives": 26486,
            "prevalence": 0.21649147471841232,
            "ap": 0.24833007922974734,
            "auroc": 0.6096886114791633
          },
          "coniferous_forest": {
            "positives": 39554,
            "prevalence": 0.3233067957038466,
            "ap": 0.6422001137870713,
            "auroc": 0.7970964351030685
          },
          "industrial_commercial_units": {
            "positives": 2726,
            "prevalence": 0.022281800199440913,
            "ap": 0.22099647423916488,
            "auroc": 0.8520227524606008
          },
          "inland_waters": {
            "positives": 15900,
            "prevalence": 0.12996354481698844,
            "ap": 0.6169913791483067,
            "auroc": 0.8810344713080226
          },
          "inland_wetlands": {
            "positives": 5056,
            "prevalence": 0.0413267724902323,
            "ap": 0.14947754213519263,
            "auroc": 0.7437969547945334
          },
          "marine_waters": {
            "positives": 14064,
            "prevalence": 0.11495643360415883,
            "ap": 0.41497809164431293,
            "auroc": 0.6439348045526262
          },
          "mixed_forest": {
            "positives": 42705,
            "prevalence": 0.34906246423959064,
            "ap": 0.6110919670131725,
            "auroc": 0.7684456891039121
          },
          "moors_heathland_sclerophyllous_vegetation": {
            "positives": 3806,
            "prevalence": 0.031109512677575977,
            "ap": 0.09946587776725221,
            "auroc": 0.7225093426987386
          },
          "natural_grassland_sparse_vegetation": {
            "positives": 2687,
            "prevalence": 0.021963021693286035,
            "ap": 0.03707585360966133,
            "auroc": 0.65062777946589
          },
          "pastures": {
            "positives": 25915,
            "prevalence": 0.2118242304359909,
            "ap": 0.554245598162242,
            "auroc": 0.7869961657876963
          },
          "permanent_crops": {
            "positives": 8688,
            "prevalence": 0.07101404260188651,
            "ap": 0.07950672139254811,
            "auroc": 0.5688205676206493
          },
          "transitional_woodland_shrub": {
            "positives": 36429,
            "prevalence": 0.2977636461722058,
            "ap": 0.45025846933136915,
            "auroc": 0.7029293438069408
          },
          "urban_fabric": {
            "positives": 17308,
            "prevalence": 0.14147226626996454,
            "ap": 0.6192617126067916,
            "auroc": 0.8676989041623855
          }
        }
      }
    }
  },
  "NIH ChestX-ray14": {
    "siglip2": {
      "model": "SigLIP2 ViT-L/16-256",
      "ranking": {
        "num_images": 25596,
        "num_classes": 14,
        "mean_labels": 1.062900453195812,
        "macro_ap": 0.09695511250054259,
        "micro_ap": 0.10714126701376775,
        "macro_auroc": 0.56857270108819,
        "ap_classes": 14,
        "auroc_classes": 14,
        "per_class": {
          "Atelectasis": {
            "positives": 3279,
            "prevalence": 0.12810595405532113,
            "ap": 0.13252736486079209,
            "auroc": 0.5070055946065237
          },
          "Cardiomegaly": {
            "positives": 1069,
            "prevalence": 0.04176433817784029,
            "ap": 0.09882707781470505,
            "auroc": 0.6144581010606551
          },
          "Consolidation": {
            "positives": 1815,
            "prevalence": 0.07090951711204876,
            "ap": 0.08752402293842668,
            "auroc": 0.5765977144751644
          },
          "Edema": {
            "positives": 925,
            "prevalence": 0.03613845913423973,
            "ap": 0.05032969399029191,
            "auroc": 0.6038178975862896
          },
          "Effusion": {
            "positives": 4658,
            "prevalence": 0.18198155961869042,
            "ap": 0.19368288871943862,
            "auroc": 0.5093855887514471
          },
          "Emphysema": {
            "positives": 1093,
            "prevalence": 0.042701984685107046,
            "ap": 0.0453372600347245,
            "auroc": 0.5098422700000623
          },
          "Fibrosis": {
            "positives": 435,
            "prevalence": 0.016994842944210033,
            "ap": 0.02920908385297373,
            "auroc": 0.5928729327955552
          },
          "Hernia": {
            "positives": 86,
            "prevalence": 0.0033598999843725583,
            "ap": 0.0053721635269742786,
            "auroc": 0.6395139161113289
          },
          "Infiltration": {
            "positives": 6112,
            "prevalence": 0.23878731051726831,
            "ap": 0.3204687156991054,
            "auroc": 0.6008805360567028
          },
          "Mass": {
            "positives": 1748,
            "prevalence": 0.06829192061259572,
            "ap": 0.11502425997170782,
            "auroc": 0.6084758077856938
          },
          "Nodule": {
            "positives": 1623,
            "prevalence": 0.06340834505391467,
            "ap": 0.07919812256609568,
            "auroc": 0.5686753034625445
          },
          "Pleural_Thickening": {
            "positives": 1143,
            "prevalence": 0.044655414908579466,
            "ap": 0.0516160491502138,
            "auroc": 0.5322026338741355
          },
          "Pneumonia": {
            "positives": 555,
            "prevalence": 0.021683075480543833,
            "ap": 0.030910043752403873,
            "auroc": 0.5914515689764281
          },
          "Pneumothorax": {
            "positives": 2665,
            "prevalence": 0.10411783091107986,
            "ap": 0.11734482812974295,
            "auroc": 0.5048379496921305
          }
        }
      },
      "decisions": {
        "micro_f1": 0.23748044316114836,
        "macro_f1": 0.14410355584080817,
        "micro_precision": 0.14358445873196543,
        "micro_recall": 0.6862456810997574,
        "exact_match": 0.0030473511486169714,
        "mean_predicted_labels": 5.08001250195343,
        "per_class_f1": [
          0.2270951392066453,
          0.13696808510638298,
          0.1439292414902171,
          0.08519511468573131,
          0.3077543883247024,
          0.06608695652173913,
          0.0639269406392694,
          0.00558659217877095,
          0.3855063291139241,
          0.1640767824497258,
          0.1270718232044199,
          0.09154746423927178,
          0.024067388688327317,
          0.18863753592218702
        ]
      },
      "calibration_images": 8668,
      "configuration": {
        "dataset": "nih-chest-xray-14",
        "model": "ViT-L-16-SigLIP2-256",
        "pretrained": "webli",
        "device": "cuda:0",
        "amp": true,
        "batch_size": 128,
        "workers": 6,
        "download_data": true,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 0,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/nih-siglip2-full"
      },
      "evaluation_patients": 2797
    },
    "pecore": {
      "model": "PE-Core-L/14-336",
      "ranking": {
        "num_images": 25596,
        "num_classes": 14,
        "mean_labels": 1.062900453195812,
        "macro_ap": 0.0862675091866708,
        "micro_ap": 0.08814599965933541,
        "macro_auroc": 0.5444101113231284,
        "ap_classes": 14,
        "auroc_classes": 14,
        "per_class": {
          "Atelectasis": {
            "positives": 3279,
            "prevalence": 0.12810595405532113,
            "ap": 0.14080124503429736,
            "auroc": 0.5165327722642618
          },
          "Cardiomegaly": {
            "positives": 1069,
            "prevalence": 0.04176433817784029,
            "ap": 0.0704806680145038,
            "auroc": 0.5455595164535463
          },
          "Consolidation": {
            "positives": 1815,
            "prevalence": 0.07090951711204876,
            "ap": 0.059370278694601056,
            "auroc": 0.4391696591359424
          },
          "Edema": {
            "positives": 925,
            "prevalence": 0.03613845913423973,
            "ap": 0.04486800270321924,
            "auroc": 0.5567000099690302
          },
          "Effusion": {
            "positives": 4658,
            "prevalence": 0.18198155961869042,
            "ap": 0.1758445655197401,
            "auroc": 0.47557243469350985
          },
          "Emphysema": {
            "positives": 1093,
            "prevalence": 0.042701984685107046,
            "ap": 0.0479430955072635,
            "auroc": 0.5150474507313348
          },
          "Fibrosis": {
            "positives": 435,
            "prevalence": 0.016994842944210033,
            "ap": 0.035571162824651115,
            "auroc": 0.6509313126910969
          },
          "Hernia": {
            "positives": 86,
            "prevalence": 0.0033598999843725583,
            "ap": 0.004533618405985192,
            "auroc": 0.6214523716189729
          },
          "Infiltration": {
            "positives": 6112,
            "prevalence": 0.23878731051726831,
            "ap": 0.20994870017961859,
            "auroc": 0.4519425079015027
          },
          "Mass": {
            "positives": 1748,
            "prevalence": 0.06829192061259572,
            "ap": 0.07464092814672699,
            "auroc": 0.5351279643309227
          },
          "Nodule": {
            "positives": 1623,
            "prevalence": 0.06340834505391467,
            "ap": 0.09421731958197925,
            "auroc": 0.5896489398797101
          },
          "Pleural_Thickening": {
            "positives": 1143,
            "prevalence": 0.044655414908579466,
            "ap": 0.0582941635825982,
            "auroc": 0.5578546792802905
          },
          "Pneumonia": {
            "positives": 555,
            "prevalence": 0.021683075480543833,
            "ap": 0.0343486953822729,
            "auroc": 0.6052236134541154
          },
          "Pneumothorax": {
            "positives": 2665,
            "prevalence": 0.10411783091107986,
            "ap": 0.15688268503593392,
            "auroc": 0.5609783261195611
          }
        }
      },
      "decisions": {
        "micro_f1": 0.21777945138023927,
        "macro_f1": 0.1336960517620505,
        "micro_precision": 0.1292629754691938,
        "micro_recall": 0.69087701242373,
        "exact_match": 0.0012501953430223472,
        "mean_predicted_labels": 5.680926707298015,
        "per_class_f1": [
          0.22669410246573432,
          0.0980891719745223,
          0.13314738696418085,
          0.08106708106708106,
          0.3077893618434159,
          0.04230317273795535,
          0.052109181141439205,
          0.0,
          0.38562215477996964,
          0.1280751729495764,
          0.1318622174381055,
          0.10153256704980843,
          0.06629834254143646,
          0.11715481171548117
        ]
      },
      "calibration_images": 8668,
      "configuration": {
        "dataset": "nih-chest-xray-14",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "device": "cuda:1",
        "amp": true,
        "batch_size": 128,
        "workers": 6,
        "download_data": true,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 0,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/nih-pecore-full"
      },
      "evaluation_patients": 2797
    },
    "biomedclip": {
      "model": "BiomedCLIP PubMedBERT ViT-B/16-224",
      "ranking": {
        "num_images": 25596,
        "num_classes": 14,
        "mean_labels": 1.062900453195812,
        "macro_ap": 0.12284179832772608,
        "micro_ap": 0.09326746102776766,
        "macro_auroc": 0.6296093627322998,
        "ap_classes": 14,
        "auroc_classes": 14,
        "per_class": {
          "Atelectasis": {
            "positives": 3279,
            "prevalence": 0.12810595405532113,
            "ap": 0.1378229241017769,
            "auroc": 0.5325375607890536
          },
          "Cardiomegaly": {
            "positives": 1069,
            "prevalence": 0.04176433817784029,
            "ap": 0.22209237076474672,
            "auroc": 0.8106703431353385
          },
          "Consolidation": {
            "positives": 1815,
            "prevalence": 0.07090951711204876,
            "ap": 0.09374198870084516,
            "auroc": 0.5906639013041756
          },
          "Edema": {
            "positives": 925,
            "prevalence": 0.03613845913423973,
            "ap": 0.07126395906611517,
            "auroc": 0.6829149225428257
          },
          "Effusion": {
            "positives": 4658,
            "prevalence": 0.18198155961869042,
            "ap": 0.3587834052453584,
            "auroc": 0.7061948388300185
          },
          "Emphysema": {
            "positives": 1093,
            "prevalence": 0.042701984685107046,
            "ap": 0.0521105124043828,
            "auroc": 0.5522879193350076
          },
          "Fibrosis": {
            "positives": 435,
            "prevalence": 0.016994842944210033,
            "ap": 0.04276980619756387,
            "auroc": 0.6838457803012964
          },
          "Hernia": {
            "positives": 86,
            "prevalence": 0.0033598999843725583,
            "ap": 0.010691603730977696,
            "auroc": 0.7731120035006791
          },
          "Infiltration": {
            "positives": 6112,
            "prevalence": 0.23878731051726831,
            "ap": 0.22777452873221066,
            "auroc": 0.4680697029163948
          },
          "Mass": {
            "positives": 1748,
            "prevalence": 0.06829192061259572,
            "ap": 0.12351276197604985,
            "auroc": 0.58482165989098
          },
          "Nodule": {
            "positives": 1623,
            "prevalence": 0.06340834505391467,
            "ap": 0.08066296223425529,
            "auroc": 0.5592764313128096
          },
          "Pleural_Thickening": {
            "positives": 1143,
            "prevalence": 0.044655414908579466,
            "ap": 0.06545675855822772,
            "auroc": 0.5871831580492998
          },
          "Pneumonia": {
            "positives": 555,
            "prevalence": 0.021683075480543833,
            "ap": 0.039901758435565336,
            "auroc": 0.6319572477713127
          },
          "Pneumothorax": {
            "positives": 2665,
            "prevalence": 0.10411783091107986,
            "ap": 0.19319983644008948,
            "auroc": 0.6509956085730068
          }
        }
      },
      "decisions": {
        "micro_f1": 0.22191738201977063,
        "macro_f1": 0.17669775854563302,
        "micro_precision": 0.13738880717237514,
        "micro_recall": 0.5767845328236418,
        "exact_match": 0.005157055789967183,
        "mean_predicted_labels": 4.462259728082513,
        "per_class_f1": [
          0.23734229067671636,
          0.2920395666509656,
          0.15346663204362504,
          0.1290704558910598,
          0.407717510010921,
          0.08385576807890428,
          0.07323232323232323,
          0.02867383512544803,
          0.38277078411701887,
          0.1363825363825364,
          0.1279947489333771,
          0.12136862491930278,
          0.07469313136589188,
          0.22516041221077193
        ]
      },
      "calibration_images": 8668,
      "configuration": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "model": "hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
        "pretrained": null,
        "score": "cosine",
        "prompt_family": "baseline",
        "device": "cuda:0",
        "batch_size": 128,
        "workers": 4,
        "amp": true,
        "limit": 0,
        "calibration_limit": 0,
        "save_features": true,
        "calibrate": "per-class"
      },
      "evaluation_patients": 2797
    }
  },
  "Plant Pathology 2021": {
    "clip": {
      "model": "CLIP ViT-B/16 LAION-2B",
      "ranking": {
        "num_images": 1864,
        "num_classes": 6,
        "mean_labels": 1.0836909871244635,
        "macro_ap": 0.3031869636137696,
        "micro_ap": 0.2683674073880171,
        "macro_auroc": 0.6742417630914659,
        "ap_classes": 6,
        "auroc_classes": 6,
        "per_class": {
          "complex": {
            "positives": 215,
            "prevalence": 0.11534334763948498,
            "ap": 0.22303890879998794,
            "auroc": 0.7036202349556462
          },
          "frog_eye_leaf_spot": {
            "positives": 435,
            "prevalence": 0.23336909871244635,
            "ap": 0.32613070799820665,
            "auroc": 0.5853776051092717
          },
          "healthy": {
            "positives": 463,
            "prevalence": 0.24839055793991416,
            "ap": 0.5070558308318075,
            "auroc": 0.8211151244945372
          },
          "powdery_mildew": {
            "positives": 127,
            "prevalence": 0.06813304721030043,
            "ap": 0.2848632829319899,
            "auroc": 0.7598447862410981
          },
          "rust": {
            "positives": 208,
            "prevalence": 0.11158798283261803,
            "ap": 0.14620694104172055,
            "auroc": 0.6007104120215534
          },
          "scab": {
            "positives": 572,
            "prevalence": 0.30686695278969955,
            "ap": 0.3318261100789052,
            "auroc": 0.5747824157266882
          }
        }
      },
      "decisions": {
        "micro_f1": 0.40764331210191085,
        "macro_f1": 0.3879939212292424,
        "micro_precision": 0.28543111992071357,
        "micro_recall": 0.7128712871287128,
        "exact_match": 0.08744635193133048,
        "mean_predicted_labels": 2.7065450643776825,
        "per_class_f1": [
          0.2842377260981912,
          0.3755795981452859,
          0.5921501706484642,
          0.35668789808917195,
          0.22988505747126436,
          0.4894230769230769
        ]
      },
      "calibration_images": 1024,
      "configuration": {
        "dataset": "plant-pathology-2021",
        "model": "ViT-B-16",
        "pretrained": "laion2b_s34b_b88k",
        "device": "cuda:0",
        "amp": true,
        "batch_size": 32,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 1024,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/plant-clip-full"
      },
      "evaluation_patients": null
    },
    "siglip2": {
      "model": "SigLIP2 ViT-L/16-256",
      "ranking": {
        "num_images": 1864,
        "num_classes": 6,
        "mean_labels": 1.0836909871244635,
        "macro_ap": 0.43963215639678793,
        "micro_ap": 0.40099820676844383,
        "macro_auroc": 0.749814443804067,
        "ap_classes": 6,
        "auroc_classes": 6,
        "per_class": {
          "complex": {
            "positives": 215,
            "prevalence": 0.11534334763948498,
            "ap": 0.16919842078764247,
            "auroc": 0.6754185059303031
          },
          "frog_eye_leaf_spot": {
            "positives": 435,
            "prevalence": 0.23336909871244635,
            "ap": 0.4728028701907264,
            "auroc": 0.7818665894484528
          },
          "healthy": {
            "positives": 463,
            "prevalence": 0.24839055793991416,
            "ap": 0.466859461191777,
            "auroc": 0.7374198929182025
          },
          "powdery_mildew": {
            "positives": 127,
            "prevalence": 0.06813304721030043,
            "ap": 0.6409557930867624,
            "auroc": 0.8264724681435547
          },
          "rust": {
            "positives": 208,
            "prevalence": 0.11158798283261803,
            "ap": 0.5226994811685083,
            "auroc": 0.8532071604422149
          },
          "scab": {
            "positives": 572,
            "prevalence": 0.30686695278969955,
            "ap": 0.3652769119553108,
            "auroc": 0.6245020459416744
          }
        }
      },
      "decisions": {
        "micro_f1": 0.4728140446565536,
        "macro_f1": 0.4827740356481178,
        "micro_precision": 0.36054068105016895,
        "micro_recall": 0.6866336633663367,
        "exact_match": 0.20815450643776823,
        "mean_predicted_labels": 2.063841201716738,
        "per_class_f1": [
          0.2802802802802803,
          0.5333333333333333,
          0.5054624908958485,
          0.6190476190476191,
          0.4448462929475588,
          0.5136741973840666
        ]
      },
      "calibration_images": 1024,
      "configuration": {
        "dataset": "plant-pathology-2021",
        "model": "ViT-L-16-SigLIP2-256",
        "pretrained": "webli",
        "device": "cuda:0",
        "amp": true,
        "batch_size": 32,
        "score": "logit",
        "threshold": 0.0,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 1024,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/plant-siglip2-full"
      },
      "evaluation_patients": null
    },
    "pecore": {
      "model": "PE-Core-L/14-336",
      "ranking": {
        "num_images": 1864,
        "num_classes": 6,
        "mean_labels": 1.0836909871244635,
        "macro_ap": 0.5161484477051028,
        "micro_ap": 0.4732058842143387,
        "macro_auroc": 0.8066513042810394,
        "ap_classes": 6,
        "auroc_classes": 6,
        "per_class": {
          "complex": {
            "positives": 215,
            "prevalence": 0.11534334763948498,
            "ap": 0.24098248511269102,
            "auroc": 0.7208893339162565
          },
          "frog_eye_leaf_spot": {
            "positives": 435,
            "prevalence": 0.23336909871244635,
            "ap": 0.39031666408080135,
            "auroc": 0.7361662765538155
          },
          "healthy": {
            "positives": 463,
            "prevalence": 0.24839055793991416,
            "ap": 0.7180293708127171,
            "auroc": 0.8753420497238166
          },
          "powdery_mildew": {
            "positives": 127,
            "prevalence": 0.06813304721030043,
            "ap": 0.7395017710939373,
            "auroc": 0.9127285255146215
          },
          "rust": {
            "positives": 208,
            "prevalence": 0.11158798283261803,
            "ap": 0.5099566356124033,
            "auroc": 0.8611735298216276
          },
          "scab": {
            "positives": 572,
            "prevalence": 0.30686695278969955,
            "ap": 0.4981037595180665,
            "auroc": 0.7336081101560978
          }
        }
      },
      "decisions": {
        "micro_f1": 0.5132773109243698,
        "macro_f1": 0.5360768659582439,
        "micro_precision": 0.3885496183206107,
        "micro_recall": 0.755940594059406,
        "exact_match": 0.23122317596566525,
        "mean_predicted_labels": 2.1083690987124464,
        "per_class_f1": [
          0.2971818958155423,
          0.5026985350809561,
          0.6574074074074074,
          0.7217391304347827,
          0.46153846153846156,
          0.5758957654723127
        ]
      },
      "calibration_images": 1024,
      "configuration": {
        "dataset": "plant-pathology-2021",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "device": "cuda:0",
        "amp": true,
        "batch_size": 128,
        "workers": 0,
        "download_data": true,
        "score": "cosine",
        "threshold": null,
        "calibrate": "per-class",
        "limit": 0,
        "calibration_limit": 1024,
        "shuffle_buffer": 0,
        "seed": 42,
        "revision": "e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46",
        "cache_dir": null,
        "prompts": null,
        "output": "results/multilabel/plant-pecore-full"
      },
      "evaluation_patients": null
    }
  }
}

linear-probe-summary.json

{
  "results": {
    "plant": {
      "method": "Independent L2-regularized logistic heads on standardized, frozen global image embeddings",
      "features": {
        "dataset": "plant-pathology-2021",
        "revision": "e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "feature_kind": "normalized projected global image embedding",
        "image_amp": "bfloat16",
        "features_dtype": "float32",
        "image_augmentation": false,
        "preprocess": "Compose(\n    Resize(size=(336, 336), interpolation=bilinear, max_size=None, antialias=True)\n    MaybeConvertMode()\n    MaybeToTensor()\n    Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))\n)",
        "paths": {
          "train": "results/multilabel/linear-probe/plant/train.npz",
          "calibration": "results/multilabel/linear-probe/plant/calibration.npz",
          "evaluation": "results/multilabel/linear-probe/plant/evaluation.npz"
        },
        "counts": {
          "train": 15744,
          "calibration": 1024,
          "evaluation": 1864
        },
        "classnames": [
          "complex",
          "frog_eye_leaf_spot",
          "healthy",
          "powdery_mildew",
          "rust",
          "scab"
        ],
        "verification": "All counts, label coverage, vocabulary, pairwise ID/patient disjointness, and evaluation targets checked."
      },
      "selection": {
        "selected_at_utc": "2026-09-28T04:14:12.705391+00:00",
        "selection_metric": "calibration macro AP",
        "selected_l2": 0.01,
        "candidates": [
          {
            "l2": 1.0,
            "optimization": {
              "iterations": 65,
              "function_evaluations": 67,
              "objective": 0.2013119843936992,
              "max_abs_gradient": 5.8294268787985326e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.8940136231857071,
            "calibration_macro_auroc": 0.9709971035999118
          },
          {
            "l2": 0.1,
            "optimization": {
              "iterations": 109,
              "function_evaluations": 115,
              "objective": 0.13611361812023848,
              "max_abs_gradient": 2.4830160847235695e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.9212110783371572,
            "calibration_macro_auroc": 0.9797261495042885
          },
          {
            "l2": 0.01,
            "optimization": {
              "iterations": 213,
              "function_evaluations": 252,
              "objective": 0.10268108299469493,
              "max_abs_gradient": 3.1384996772214975e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.9337047307705499,
            "calibration_macro_auroc": 0.9835769734908203
          },
          {
            "l2": 0.001,
            "optimization": {
              "iterations": 478,
              "function_evaluations": 584,
              "objective": 0.08177726806668435,
              "max_abs_gradient": 2.648081407795387e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.9320870713988949,
            "calibration_macro_auroc": 0.9832705132053391
          },
          {
            "l2": 0.0001,
            "optimization": {
              "iterations": 1328,
              "function_evaluations": 1485,
              "objective": 0.06897256386039924,
              "max_abs_gradient": 3.206436764809042e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.9249130775518628,
            "calibration_macro_auroc": 0.9809199118365094
          }
        ],
        "threshold_method": "per-class calibration F1",
        "training_images": 15744,
        "calibration_images": 1024,
        "training_positives": [
          1810,
          3685,
          3893,
          1079,
          1746,
          4834
        ]
      },
      "optimization_dtype": "float64",
      "ranking": {
        "num_images": 1864,
        "num_classes": 6,
        "mean_labels": 1.0836909871244635,
        "macro_ap": 0.9365420750339933,
        "micro_ap": 0.9592589612822102,
        "macro_auroc": 0.9825908947860517,
        "ap_classes": 6,
        "auroc_classes": 6,
        "per_class": {
          "complex": {
            "positives": 215,
            "prevalence": 0.11534334763948498,
            "ap": 0.7675441715844553,
            "auroc": 0.9563512770248354
          },
          "frog_eye_leaf_spot": {
            "positives": 435,
            "prevalence": 0.23336909871244635,
            "ap": 0.936145225069895,
            "auroc": 0.9718845265960442
          },
          "healthy": {
            "positives": 463,
            "prevalence": 0.24839055793991416,
            "ap": 0.9967185637935417,
            "auroc": 0.9986217804931067
          },
          "powdery_mildew": {
            "positives": 127,
            "prevalence": 0.06813304721030043,
            "ap": 0.992436260873703,
            "auroc": 0.9992384371642664
          },
          "rust": {
            "positives": 208,
            "prevalence": 0.11158798283261803,
            "ap": 0.9704711630563027,
            "auroc": 0.9947800538833147
          },
          "scab": {
            "positives": 572,
            "prevalence": 0.30686695278969955,
            "ap": 0.9559370658260627,
            "auroc": 0.9746692935547425
          }
        }
      },
      "calibrated_thresholds": {
        "thresholds": [
          -0.3004848773593696,
          -0.22185962513624435,
          0.5799971202282297,
          0.07739185962078743,
          -1.9179767774503236,
          -0.35148753860961013
        ],
        "global_fallback_classes": [],
        "metrics": {
          "micro_f1": 0.8947238048055487,
          "macro_f1": 0.8904240652412784,
          "micro_precision": 0.8953891918691126,
          "micro_recall": 0.8940594059405941,
          "exact_match": 0.8369098712446352,
          "mean_predicted_labels": 1.0820815450643777,
          "per_class_f1": [
            0.7412587412587412,
            0.8611435239206534,
            0.982646420824295,
            0.968,
            0.9017857142857143,
            0.887709991158267
          ]
        }
      },
      "fixed_probability_0_5": {
        "micro_f1": 0.8912547528517111,
        "macro_f1": 0.8831689670818008,
        "micro_precision": 0.9132467532467532,
        "micro_recall": 0.8702970297029703,
        "exact_match": 0.8363733905579399,
        "mean_predicted_labels": 1.032725321888412,
        "per_class_f1": [
          0.7052896725440806,
          0.8581644815256257,
          0.9828326180257511,
          0.968,
          0.8990384615384616,
          0.8856885688568857
        ]
      },
      "verification": "Training, calibration and evaluation image IDs/patients are pairwise disjoint.",
      "zero_shot_same_encoder": {
        "ranking": {
          "num_images": 1864,
          "num_classes": 6,
          "mean_labels": 1.0836909871244635,
          "macro_ap": 0.5160859969186128,
          "micro_ap": 0.4731163267704931,
          "macro_auroc": 0.8066397156766358,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 215,
              "prevalence": 0.11534334763948498,
              "ap": 0.24061583044138377,
              "auroc": 0.720614325807043
            },
            "frog_eye_leaf_spot": {
              "positives": 435,
              "prevalence": 0.23336909871244635,
              "ap": 0.3902228208570089,
              "auroc": 0.736338408822181
            },
            "healthy": {
              "positives": 463,
              "prevalence": 0.24839055793991416,
              "ap": 0.7181315900294756,
              "auroc": 0.8753952360470691
            },
            "powdery_mildew": {
              "positives": 127,
              "prevalence": 0.06813304721030043,
              "ap": 0.7390018848405069,
              "auroc": 0.9128418533175581
            },
            "rust": {
              "positives": 208,
              "prevalence": 0.11158798283261803,
              "ap": 0.5105013377142232,
              "auroc": 0.861264980490524
            },
            "scab": {
              "positives": 572,
              "prevalence": 0.30686695278969955,
              "ap": 0.49804251762907853,
              "auroc": 0.73338348957544
            }
          }
        },
        "calibrated_metrics": {
          "micro_f1": 0.5120562625586068,
          "macro_f1": 0.534610667366547,
          "micro_precision": 0.3868927125506073,
          "micro_recall": 0.7569306930693069,
          "exact_match": 0.23015021459227467,
          "mean_predicted_labels": 2.1201716738197427,
          "per_class_f1": [
            0.29642248722316866,
            0.5003849114703618,
            0.6556473829201102,
            0.721030042918455,
            0.45865834633385333,
            0.5755208333333334
          ]
        }
      }
    },
    "bigearth": {
      "method": "Independent L2-regularized logistic heads on standardized, frozen global image embeddings",
      "features": {
        "dataset": "bigearthnet-v2-rgb",
        "revision": "edb997737d3888abe5257c71d8e4fabc3c8ec34d",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "feature_kind": "normalized projected global image embedding",
        "image_amp": "bfloat16",
        "features_dtype": "float32",
        "image_augmentation": false,
        "preprocess": "Compose(\n    Resize(size=(336, 336), interpolation=bilinear, max_size=None, antialias=True)\n    MaybeConvertMode()\n    MaybeToTensor()\n    Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))\n)",
        "paths": {
          "train": "results/multilabel/linear-probe/bigearth/train.npz",
          "calibration": "results/multilabel/linear-probe/bigearth/calibration.npz",
          "evaluation": "results/multilabel/linear-probe/bigearth/evaluation.npz"
        },
        "counts": {
          "train": 237871,
          "calibration": 122342,
          "evaluation": 119825
        },
        "classnames": [
          "agriculture_with_natural_vegetation",
          "agro_forestry_areas",
          "arable_land",
          "beaches_dunes_sands",
          "broad_leaved_forest",
          "coastal_wetlands",
          "complex_cultivation_patterns",
          "coniferous_forest",
          "industrial_commercial_units",
          "inland_waters",
          "inland_wetlands",
          "marine_waters",
          "mixed_forest",
          "moors_heathland_sclerophyllous_vegetation",
          "natural_grassland_sparse_vegetation",
          "pastures",
          "permanent_crops",
          "transitional_woodland_shrub",
          "urban_fabric"
        ],
        "verification": "All counts, label coverage, vocabulary, pairwise ID/patient disjointness, and evaluation targets checked."
      },
      "selection": {
        "selected_at_utc": "2026-09-28T04:35:33.142528+00:00",
        "selection_metric": "calibration macro AP",
        "selected_l2": 0.001,
        "candidates": [
          {
            "l2": 1.0,
            "optimization": {
              "iterations": 180,
              "function_evaluations": 200,
              "objective": 0.2311528721336742,
              "max_abs_gradient": 4.4438217737590514e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.6565262129739392,
            "calibration_macro_auroc": 0.9158266731509324
          },
          {
            "l2": 0.1,
            "optimization": {
              "iterations": 1000,
              "function_evaluations": 1382,
              "objective": 0.19349186739036375,
              "max_abs_gradient": 2.9172944221231047e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.7033993875310048,
            "calibration_macro_auroc": 0.9297745423578123
          },
          {
            "l2": 0.01,
            "optimization": {
              "iterations": 893,
              "function_evaluations": 1353,
              "objective": 0.17421740191147067,
              "max_abs_gradient": 2.1640760644298215e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.7333360724718454,
            "calibration_macro_auroc": 0.9372907076765381
          },
          {
            "l2": 0.001,
            "optimization": {
              "iterations": 1273,
              "function_evaluations": 1876,
              "objective": 0.16439414580623884,
              "max_abs_gradient": 1.3639702568906743e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.7433959424963138,
            "calibration_macro_auroc": 0.9394873930961062
          },
          {
            "l2": 0.0001,
            "optimization": {
              "iterations": 1500,
              "function_evaluations": 1769,
              "objective": 0.16035664198596472,
              "max_abs_gradient": 5.89075172281509e-08,
              "max_iter": 1500,
              "hit_iteration_limit": true
            },
            "calibration_macro_ap": 0.7409445230168761,
            "calibration_macro_auroc": 0.9387398995365824
          }
        ],
        "threshold_method": "per-class calibration F1",
        "training_images": 237871,
        "calibration_images": 122342,
        "training_positives": [
          60127,
          15082,
          88236,
          738,
          64164,
          670,
          51034,
          76344,
          6398,
          30466,
          11344,
          35914,
          78791,
          6329,
          6984,
          42968,
          15190,
          64198,
          33626
        ]
      },
      "optimization_dtype": "float64",
      "ranking": {
        "num_images": 119825,
        "num_classes": 19,
        "mean_labels": 2.9950093886918423,
        "macro_ap": 0.6903262485100645,
        "micro_ap": 0.8302673137408155,
        "macro_auroc": 0.9363427096090994,
        "ap_classes": 19,
        "auroc_classes": 19,
        "per_class": {
          "agriculture_with_natural_vegetation": {
            "positives": 29846,
            "prevalence": 0.24907990819945755,
            "ap": 0.6876630672879841,
            "auroc": 0.8672259352567391
          },
          "agro_forestry_areas": {
            "positives": 9942,
            "prevalence": 0.08297099937408721,
            "ap": 0.8439762985241217,
            "auroc": 0.9849198309616249
          },
          "arable_land": {
            "positives": 50052,
            "prevalence": 0.4177091591904861,
            "ap": 0.9209199188809163,
            "auroc": 0.9373730436909327
          },
          "beaches_dunes_sands": {
            "positives": 152,
            "prevalence": 0.0012685165866889213,
            "ap": 0.1850746178510647,
            "auroc": 0.9701351203960618
          },
          "broad_leaved_forest": {
            "positives": 36377,
            "prevalence": 0.3035843939077822,
            "ap": 0.7905985555575465,
            "auroc": 0.8923639992666514
          },
          "coastal_wetlands": {
            "positives": 117,
            "prevalence": 0.0009764239515960776,
            "ap": 0.4174664348284451,
            "auroc": 0.9870388315270863
          },
          "complex_cultivation_patterns": {
            "positives": 22078,
            "prevalence": 0.18425203421656583,
            "ap": 0.6680229596599042,
            "auroc": 0.904185554552585
          },
          "coniferous_forest": {
            "positives": 39043,
            "prevalence": 0.3258335071979971,
            "ap": 0.8823278590271515,
            "auroc": 0.934771810467771
          },
          "industrial_commercial_units": {
            "positives": 2018,
            "prevalence": 0.01684122678906739,
            "ap": 0.5449077332886364,
            "auroc": 0.9601673717346381
          },
          "inland_waters": {
            "positives": 16846,
            "prevalence": 0.14058835802211558,
            "ap": 0.8997442618178646,
            "auroc": 0.9695985224177451
          },
          "inland_wetlands": {
            "positives": 4519,
            "prevalence": 0.03771333194241602,
            "ap": 0.5090878310757233,
            "auroc": 0.9186798745546773
          },
          "marine_waters": {
            "positives": 11854,
            "prevalence": 0.09892760275401627,
            "ap": 0.9934041682654278,
            "auroc": 0.9987010029814837
          },
          "mixed_forest": {
            "positives": 44284,
            "prevalence": 0.36957229292718546,
            "ap": 0.8517579555080745,
            "auroc": 0.9095876682196739
          },
          "moors_heathland_sclerophyllous_vegetation": {
            "positives": 3759,
            "prevalence": 0.03137074900897142,
            "ap": 0.5944636812240919,
            "auroc": 0.967865743631834
          },
          "natural_grassland_sparse_vegetation": {
            "positives": 2211,
            "prevalence": 0.018451909034007927,
            "ap": 0.3893178507237084,
            "auroc": 0.9215434405905691
          },
          "pastures": {
            "positives": 26722,
            "prevalence": 0.2230085541414563,
            "ap": 0.8280912035302525,
            "auroc": 0.9234449326375739
          },
          "permanent_crops": {
            "positives": 5710,
            "prevalence": 0.04765282703943251,
            "ap": 0.5203347109251605,
            "auroc": 0.9237256284236575
          },
          "transitional_woodland_shrub": {
            "positives": 40523,
            "prevalence": 0.3381848529104945,
            "ap": 0.7378118209800258,
            "auroc": 0.8523272323099094
          },
          "urban_fabric": {
            "positives": 12824,
            "prevalence": 0.10702274149801794,
            "ap": 0.8512277927351268,
            "auroc": 0.9668559389516727
          }
        }
      },
      "calibrated_thresholds": {
        "thresholds": [
          -0.7953435346488544,
          -0.675692907652838,
          -0.49844729520530806,
          -0.4246697185882038,
          -0.5561321568235611,
          -1.85050052815288,
          -0.8053591230493233,
          -0.3196769817340095,
          -0.8350214498436301,
          -0.28681013937177546,
          -0.8717769893604652,
          0.40135205604444213,
          -0.5479820450729864,
          -0.9376937127466443,
          -0.8901026374614212,
          -0.748414362469024,
          -0.8034404294656925,
          -0.8781720086036588,
          -0.3246468197674397
        ],
        "global_fallback_classes": [],
        "metrics": {
          "micro_f1": 0.7445099414217999,
          "macro_f1": 0.6462607087109241,
          "micro_precision": 0.7064339978037206,
          "micro_recall": 0.7869242108020297,
          "exact_match": 0.25564781973711664,
          "mean_predicted_labels": 3.336257041518882,
          "per_class_f1": [
            0.6463948262307726,
            0.7672468211741366,
            0.833341542947777,
            0.16279069767441862,
            0.7298689015579434,
            0.37572254335260113,
            0.6562568203011845,
            0.8184386877375475,
            0.5306227483273289,
            0.8153685887615663,
            0.5102291783119061,
            0.972262035241548,
            0.7832242955514733,
            0.5867698633450236,
            0.41898269533298377,
            0.7192079796422535,
            0.4971731448763251,
            0.6905475319073934,
            0.7645045632333768
          ]
        }
      },
      "fixed_probability_0_5": {
        "micro_f1": 0.7294445317939848,
        "macro_f1": 0.6180326126508459,
        "micro_precision": 0.7884009089202364,
        "micro_recall": 0.6786921424332013,
        "exact_match": 0.2745921134988525,
        "mean_predicted_labels": 2.578243271437513,
        "per_class_f1": [
          0.6035319084432318,
          0.7647000691767961,
          0.8250681654356842,
          0.14201183431952663,
          0.6985812438654413,
          0.40860215053763443,
          0.6238593633990372,
          0.8180075139266745,
          0.494596312778131,
          0.8116956850870553,
          0.46772577644196367,
          0.9712848459442465,
          0.7706434831186062,
          0.5074135090609555,
          0.30504981106149087,
          0.6979465808736208,
          0.44038871770561744,
          0.6359741879507619,
          0.7555384812395953
        ]
      },
      "verification": "Training, calibration and evaluation image IDs/patients are pairwise disjoint.",
      "zero_shot_same_encoder": {
        "ranking": {
          "num_images": 119825,
          "num_classes": 19,
          "mean_labels": 2.9950093886918423,
          "macro_ap": 0.3339164975208038,
          "micro_ap": 0.36595178963738073,
          "macro_auroc": 0.7289817113677363,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 29846,
              "prevalence": 0.24907990819945755,
              "ap": 0.26251413629146236,
              "auroc": 0.5615061968076677
            },
            "agro_forestry_areas": {
              "positives": 9942,
              "prevalence": 0.08297099937408721,
              "ap": 0.09977181396360858,
              "auroc": 0.6065920688967188
            },
            "arable_land": {
              "positives": 50052,
              "prevalence": 0.4177091591904861,
              "ap": 0.7915004736020211,
              "auroc": 0.8474299784850244
            },
            "beaches_dunes_sands": {
              "positives": 152,
              "prevalence": 0.0012685165866889213,
              "ap": 0.012395987712589438,
              "auroc": 0.7579400302227077
            },
            "broad_leaved_forest": {
              "positives": 36377,
              "prevalence": 0.3035843939077822,
              "ap": 0.38536233709626555,
              "auroc": 0.6261204379897817
            },
            "coastal_wetlands": {
              "positives": 117,
              "prevalence": 0.0009764239515960776,
              "ap": 0.08237325516367729,
              "auroc": 0.922655527310187
            },
            "complex_cultivation_patterns": {
              "positives": 22078,
              "prevalence": 0.18425203421656583,
              "ap": 0.213527004751214,
              "auroc": 0.6059437377581908
            },
            "coniferous_forest": {
              "positives": 39043,
              "prevalence": 0.3258335071979971,
              "ap": 0.6280164879508193,
              "auroc": 0.7860840736682011
            },
            "industrial_commercial_units": {
              "positives": 2018,
              "prevalence": 0.01684122678906739,
              "ap": 0.172673439686224,
              "auroc": 0.8432810470280618
            },
            "inland_waters": {
              "positives": 16846,
              "prevalence": 0.14058835802211558,
              "ap": 0.6267400322486818,
              "auroc": 0.8711684219168434
            },
            "inland_wetlands": {
              "positives": 4519,
              "prevalence": 0.03771333194241602,
              "ap": 0.12286337375863064,
              "auroc": 0.7352023761728641
            },
            "marine_waters": {
              "positives": 11854,
              "prevalence": 0.09892760275401627,
              "ap": 0.42842665007936137,
              "auroc": 0.6446721237692072
            },
            "mixed_forest": {
              "positives": 44284,
              "prevalence": 0.36957229292718546,
              "ap": 0.611654242877365,
              "auroc": 0.7534004582637761
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 3759,
              "prevalence": 0.03137074900897142,
              "ap": 0.1049148845574636,
              "auroc": 0.719409093853532
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 2211,
              "prevalence": 0.018451909034007927,
              "ap": 0.024330624947348525,
              "auroc": 0.6116698679257864
            },
            "pastures": {
              "positives": 26722,
              "prevalence": 0.2230085541414563,
              "ap": 0.6155542828489171,
              "auroc": 0.8049937002531076
            },
            "permanent_crops": {
              "positives": 5710,
              "prevalence": 0.04765282703943251,
              "ap": 0.05618839869815032,
              "auroc": 0.5727793996485402
            },
            "transitional_woodland_shrub": {
              "positives": 40523,
              "prevalence": 0.3381848529104945,
              "ap": 0.487278255102817,
              "auroc": 0.7005626140926111
            },
            "urban_fabric": {
              "positives": 12824,
              "prevalence": 0.10702274149801794,
              "ap": 0.6183277715586555,
              "auroc": 0.8792413619241775
            }
          }
        },
        "calibrated_metrics": {
          "micro_f1": 0.4698865005532029,
          "macro_f1": 0.3780723772472471,
          "micro_precision": 0.34047959385475,
          "micro_recall": 0.7579700008638057,
          "exact_match": 0.024694345921134987,
          "mean_predicted_labels": 6.667440016691008,
          "per_class_f1": [
            0.4320652384157711,
            0.18418002217518395,
            0.7352174029955094,
            0.04405286343612335,
            0.5065374033800603,
            0.14007782101167315,
            0.3560811114608156,
            0.6228724691457611,
            0.254606365159129,
            0.6370590501348302,
            0.18676519986563656,
            0.4640473097083464,
            0.6353746834540072,
            0.13067110371602883,
            0.04834926934872812,
            0.5524691897561402,
            0.10644595035924233,
            0.5746921292164912,
            0.5718105849582172
          ]
        }
      }
    },
    "nih": {
      "method": "Independent L2-regularized logistic heads on standardized, frozen global image embeddings",
      "features": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "model": "hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
        "pretrained": null,
        "feature_kind": "normalized projected global image embedding",
        "image_amp": "bfloat16",
        "features_dtype": "float32",
        "image_augmentation": false,
        "preprocess": "Compose(\n    Resize(size=224, interpolation=bicubic, max_size=None, antialias=True)\n    CenterCrop(size=(224, 224))\n    MaybeConvertMode()\n    MaybeToTensor()\n    Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])\n)",
        "paths": {
          "train": "results/multilabel/linear-probe/nih/train.npz",
          "calibration": "results/multilabel/nih-biomedclip-full/calibration.npz",
          "evaluation": "results/multilabel/nih-biomedclip-full/evaluation.npz"
        },
        "counts": {
          "train": 77856,
          "calibration": 8668,
          "evaluation": 25596
        },
        "classnames": [
          "Atelectasis",
          "Cardiomegaly",
          "Consolidation",
          "Edema",
          "Effusion",
          "Emphysema",
          "Fibrosis",
          "Hernia",
          "Infiltration",
          "Mass",
          "Nodule",
          "Pleural_Thickening",
          "Pneumonia",
          "Pneumothorax"
        ],
        "verification": "All counts, label coverage, vocabulary, pairwise ID/patient disjointness, and evaluation targets checked."
      },
      "selection": {
        "selected_at_utc": "2026-09-28T04:18:05.960915+00:00",
        "selection_metric": "calibration macro AP",
        "selected_l2": 0.0001,
        "candidates": [
          {
            "l2": 1.0,
            "optimization": {
              "iterations": 90,
              "function_evaluations": 99,
              "objective": 0.14956456205457072,
              "max_abs_gradient": 2.570056833232386e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.14268088882314015,
            "calibration_macro_auroc": 0.7512157061481386
          },
          {
            "l2": 0.1,
            "optimization": {
              "iterations": 299,
              "function_evaluations": 468,
              "objective": 0.14333498248321172,
              "max_abs_gradient": 2.0799960837558438e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.15504780288634504,
            "calibration_macro_auroc": 0.7650682719915123
          },
          {
            "l2": 0.01,
            "optimization": {
              "iterations": 211,
              "function_evaluations": 256,
              "objective": 0.140775492011489,
              "max_abs_gradient": 5.227090695952277e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.16225887955961796,
            "calibration_macro_auroc": 0.7749937553618838
          },
          {
            "l2": 0.001,
            "optimization": {
              "iterations": 357,
              "function_evaluations": 384,
              "objective": 0.13992880643872452,
              "max_abs_gradient": 4.054717586126437e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.16400972239686543,
            "calibration_macro_auroc": 0.7776670372041238
          },
          {
            "l2": 0.0001,
            "optimization": {
              "iterations": 1160,
              "function_evaluations": 1318,
              "objective": 0.13940538418117715,
              "max_abs_gradient": 7.037429574746997e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.16618340631986772,
            "calibration_macro_auroc": 0.7780253374612142
          }
        ],
        "threshold_method": "per-class calibration F1",
        "training_images": 77856,
        "calibration_images": 8668,
        "training_positives": [
          7452,
          1537,
          2567,
          1241,
          7793,
          1281,
          1126,
          127,
          12404,
          3631,
          4237,
          2018,
          788,
          2374
        ]
      },
      "optimization_dtype": "float64",
      "ranking": {
        "num_images": 25596,
        "num_classes": 14,
        "mean_labels": 1.062900453195812,
        "macro_ap": 0.19392904822519455,
        "micro_ap": 0.2974689064477158,
        "macro_auroc": 0.7491044459364204,
        "ap_classes": 14,
        "auroc_classes": 14,
        "per_class": {
          "Atelectasis": {
            "positives": 3279,
            "prevalence": 0.12810595405532113,
            "ap": 0.2592246085128337,
            "auroc": 0.7068554171809475
          },
          "Cardiomegaly": {
            "positives": 1069,
            "prevalence": 0.04176433817784029,
            "ap": 0.25953051396644716,
            "auroc": 0.8379492667308507
          },
          "Consolidation": {
            "positives": 1815,
            "prevalence": 0.07090951711204876,
            "ap": 0.14044931965827503,
            "auroc": 0.7136170818591085
          },
          "Edema": {
            "positives": 925,
            "prevalence": 0.03613845913423973,
            "ap": 0.13724401470773157,
            "auroc": 0.8188539778074049
          },
          "Effusion": {
            "positives": 4658,
            "prevalence": 0.18198155961869042,
            "ap": 0.42704588972563184,
            "auroc": 0.77522312701332
          },
          "Emphysema": {
            "positives": 1093,
            "prevalence": 0.042701984685107046,
            "ap": 0.18342861681602998,
            "auroc": 0.7863883127405389
          },
          "Fibrosis": {
            "positives": 435,
            "prevalence": 0.016994842944210033,
            "ap": 0.06012885093500553,
            "auroc": 0.785420055760443
          },
          "Hernia": {
            "positives": 86,
            "prevalence": 0.0033598999843725583,
            "ap": 0.025115196460224705,
            "auroc": 0.7892062392313092
          },
          "Infiltration": {
            "positives": 6112,
            "prevalence": 0.23878731051726831,
            "ap": 0.3775529897997421,
            "auroc": 0.6782428028945217
          },
          "Mass": {
            "positives": 1748,
            "prevalence": 0.06829192061259572,
            "ap": 0.21809627684652966,
            "auroc": 0.7428633634682508
          },
          "Nodule": {
            "positives": 1623,
            "prevalence": 0.06340834505391467,
            "ap": 0.14490433972063327,
            "auroc": 0.6667653991208378
          },
          "Pleural_Thickening": {
            "positives": 1143,
            "prevalence": 0.044655414908579466,
            "ap": 0.10816596433729521,
            "auroc": 0.720555804752517
          },
          "Pneumonia": {
            "positives": 555,
            "prevalence": 0.021683075480543833,
            "ap": 0.040505432323697216,
            "auroc": 0.6645402800668165
          },
          "Pneumothorax": {
            "positives": 2665,
            "prevalence": 0.10411783091107986,
            "ap": 0.3336146613426465,
            "auroc": 0.80098111448302
          }
        }
      },
      "calibrated_thresholds": {
        "thresholds": [
          -1.679195732534167,
          -1.3798407531875112,
          -2.1874008816710155,
          -1.923084226161544,
          -1.4318307914650925,
          -2.207793946022962,
          -2.5147176183702524,
          -1.988081156090833,
          -1.4212952609786282,
          -1.6212125701322997,
          -2.072381275614873,
          -2.8239635160655174,
          -3.2123488855832942,
          -2.1830166839136633
        ],
        "global_fallback_classes": [],
        "metrics": {
          "micro_f1": 0.33675948420855917,
          "macro_f1": 0.24853630744933833,
          "micro_precision": 0.2468070535867146,
          "micro_recall": 0.5298831140189664,
          "exact_match": 0.19733552117518363,
          "mean_predicted_labels": 2.2819971870604783,
          "per_class_f1": [
            0.3276335607335873,
            0.29807130333138515,
            0.21768172888015716,
            0.22018970189701897,
            0.462875769648678,
            0.24513246090009574,
            0.0998914223669924,
            0.0,
            0.4572695858476456,
            0.29085807809212066,
            0.21119648737650934,
            0.17247414478918058,
            0.07438303788668753,
            0.4018510225406777
          ]
        }
      },
      "fixed_probability_0_5": {
        "micro_f1": 0.11851516230298716,
        "macro_f1": 0.058968013552008966,
        "micro_precision": 0.495802870295153,
        "micro_recall": 0.06730133058884069,
        "exact_match": 0.3800203156743241,
        "mean_predicted_labels": 0.14428035630567276,
        "per_class_f1": [
          0.034047549163486936,
          0.08605851979345955,
          0.0,
          0.0064034151547491995,
          0.28062992125984254,
          0.050522648083623695,
          0.0,
          0.0,
          0.1498700943525229,
          0.12319920516641829,
          0.01218769043266301,
          0.0017482517482517483,
          0.0,
          0.0808848945731075
        ]
      },
      "verification": "Training, calibration and evaluation image IDs/patients are pairwise disjoint.",
      "zero_shot_same_encoder": {
        "ranking": {
          "num_images": 25596,
          "num_classes": 14,
          "mean_labels": 1.062900453195812,
          "macro_ap": 0.12284179832772608,
          "micro_ap": 0.09326746102776766,
          "macro_auroc": 0.6296093627322998,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 3279,
              "prevalence": 0.12810595405532113,
              "ap": 0.1378229241017769,
              "auroc": 0.5325375607890536
            },
            "Cardiomegaly": {
              "positives": 1069,
              "prevalence": 0.04176433817784029,
              "ap": 0.22209237076474672,
              "auroc": 0.8106703431353385
            },
            "Consolidation": {
              "positives": 1815,
              "prevalence": 0.07090951711204876,
              "ap": 0.09374198870084516,
              "auroc": 0.5906639013041756
            },
            "Edema": {
              "positives": 925,
              "prevalence": 0.03613845913423973,
              "ap": 0.07126395906611517,
              "auroc": 0.6829149225428257
            },
            "Effusion": {
              "positives": 4658,
              "prevalence": 0.18198155961869042,
              "ap": 0.3587834052453584,
              "auroc": 0.7061948388300185
            },
            "Emphysema": {
              "positives": 1093,
              "prevalence": 0.042701984685107046,
              "ap": 0.0521105124043828,
              "auroc": 0.5522879193350076
            },
            "Fibrosis": {
              "positives": 435,
              "prevalence": 0.016994842944210033,
              "ap": 0.04276980619756387,
              "auroc": 0.6838457803012964
            },
            "Hernia": {
              "positives": 86,
              "prevalence": 0.0033598999843725583,
              "ap": 0.010691603730977696,
              "auroc": 0.7731120035006791
            },
            "Infiltration": {
              "positives": 6112,
              "prevalence": 0.23878731051726831,
              "ap": 0.22777452873221066,
              "auroc": 0.4680697029163948
            },
            "Mass": {
              "positives": 1748,
              "prevalence": 0.06829192061259572,
              "ap": 0.12351276197604985,
              "auroc": 0.58482165989098
            },
            "Nodule": {
              "positives": 1623,
              "prevalence": 0.06340834505391467,
              "ap": 0.08066296223425529,
              "auroc": 0.5592764313128096
            },
            "Pleural_Thickening": {
              "positives": 1143,
              "prevalence": 0.044655414908579466,
              "ap": 0.06545675855822772,
              "auroc": 0.5871831580492998
            },
            "Pneumonia": {
              "positives": 555,
              "prevalence": 0.021683075480543833,
              "ap": 0.039901758435565336,
              "auroc": 0.6319572477713127
            },
            "Pneumothorax": {
              "positives": 2665,
              "prevalence": 0.10411783091107986,
              "ap": 0.19319983644008948,
              "auroc": 0.6509956085730068
            }
          }
        },
        "calibrated_metrics": {
          "micro_f1": 0.22191738201977063,
          "macro_f1": 0.17669775854563302,
          "micro_precision": 0.13738880717237514,
          "micro_recall": 0.5767845328236418,
          "exact_match": 0.005157055789967183,
          "mean_predicted_labels": 4.462259728082513,
          "per_class_f1": [
            0.23734229067671636,
            0.2920395666509656,
            0.15346663204362504,
            0.1290704558910598,
            0.407717510010921,
            0.08385576807890428,
            0.07323232323232323,
            0.02867383512544803,
            0.38277078411701887,
            0.1363825363825364,
            0.1279947489333771,
            0.12136862491930278,
            0.07469313136589188,
            0.22516041221077193
          ]
        }
      }
    },
    "nih-pecore": {
      "method": "Independent L2-regularized logistic heads on standardized, frozen global image embeddings",
      "features": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "model": "PE-Core-L-14-336",
        "pretrained": "meta",
        "feature_kind": "normalized projected global image embedding",
        "image_amp": "bfloat16",
        "features_dtype": "float32",
        "image_augmentation": false,
        "preprocess": "Compose(\n    Resize(size=(336, 336), interpolation=bilinear, max_size=None, antialias=True)\n    MaybeConvertMode()\n    MaybeToTensor()\n    Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))\n)",
        "paths": {
          "train": "results/multilabel/linear-probe/nih-pecore/train.npz",
          "calibration": "results/multilabel/prompt-comparison/nih-pecore/features.npz",
          "evaluation": "results/multilabel/linear-probe/nih-pecore/evaluation.npz"
        },
        "counts": {
          "train": 77856,
          "calibration": 8668,
          "evaluation": 25596
        },
        "classnames": [
          "Atelectasis",
          "Cardiomegaly",
          "Consolidation",
          "Edema",
          "Effusion",
          "Emphysema",
          "Fibrosis",
          "Hernia",
          "Infiltration",
          "Mass",
          "Nodule",
          "Pleural_Thickening",
          "Pneumonia",
          "Pneumothorax"
        ],
        "verification": "All counts, label coverage, vocabulary, pairwise ID/patient disjointness, and evaluation targets checked."
      },
      "selection": {
        "selected_at_utc": "2026-09-28T04:23:50.844376+00:00",
        "selection_metric": "calibration macro AP",
        "selected_l2": 0.001,
        "candidates": [
          {
            "l2": 1.0,
            "optimization": {
              "iterations": 246,
              "function_evaluations": 263,
              "objective": 0.15340738615185273,
              "max_abs_gradient": 4.4262038270761623e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.11336608582727496,
            "calibration_macro_auroc": 0.7367989294065173
          },
          {
            "l2": 0.1,
            "optimization": {
              "iterations": 266,
              "function_evaluations": 312,
              "objective": 0.1463689405278612,
              "max_abs_gradient": 2.2898804047010887e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.13709551810214943,
            "calibration_macro_auroc": 0.767410834559986
          },
          {
            "l2": 0.01,
            "optimization": {
              "iterations": 275,
              "function_evaluations": 299,
              "objective": 0.14049583025742163,
              "max_abs_gradient": 1.6543489431694351e-07,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.15588522442968292,
            "calibration_macro_auroc": 0.7838043552937277
          },
          {
            "l2": 0.001,
            "optimization": {
              "iterations": 598,
              "function_evaluations": 672,
              "objective": 0.13611297308649903,
              "max_abs_gradient": 6.606753893914182e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.16016186539292193,
            "calibration_macro_auroc": 0.7824835668459219
          },
          {
            "l2": 0.0001,
            "optimization": {
              "iterations": 1211,
              "function_evaluations": 1348,
              "objective": 0.1335683382504091,
              "max_abs_gradient": 3.96159633411908e-08,
              "max_iter": 1500,
              "hit_iteration_limit": false
            },
            "calibration_macro_ap": 0.15650062874318374,
            "calibration_macro_auroc": 0.7715659041905306
          }
        ],
        "threshold_method": "per-class calibration F1",
        "training_images": 77856,
        "calibration_images": 8668,
        "training_positives": [
          7452,
          1537,
          2567,
          1241,
          7793,
          1281,
          1126,
          127,
          12404,
          3631,
          4237,
          2018,
          788,
          2374
        ]
      },
      "optimization_dtype": "float64",
      "ranking": {
        "num_images": 25596,
        "num_classes": 14,
        "mean_labels": 1.062900453195812,
        "macro_ap": 0.19613960729503166,
        "micro_ap": 0.29583690420453,
        "macro_auroc": 0.7555634547432749,
        "ap_classes": 14,
        "auroc_classes": 14,
        "per_class": {
          "Atelectasis": {
            "positives": 3279,
            "prevalence": 0.12810595405532113,
            "ap": 0.2668052755737654,
            "auroc": 0.7173249549044779
          },
          "Cardiomegaly": {
            "positives": 1069,
            "prevalence": 0.04176433817784029,
            "ap": 0.19977908278435502,
            "auroc": 0.7883929521857568
          },
          "Consolidation": {
            "positives": 1815,
            "prevalence": 0.07090951711204876,
            "ap": 0.12930203002235074,
            "auroc": 0.7010298403603218
          },
          "Edema": {
            "positives": 925,
            "prevalence": 0.03613845913423973,
            "ap": 0.13202917228686684,
            "auroc": 0.8183043008149408
          },
          "Effusion": {
            "positives": 4658,
            "prevalence": 0.18198155961869042,
            "ap": 0.41039374328949446,
            "auroc": 0.7603540115020317
          },
          "Emphysema": {
            "positives": 1093,
            "prevalence": 0.042701984685107046,
            "ap": 0.2632504639921568,
            "auroc": 0.8235954751176163
          },
          "Fibrosis": {
            "positives": 435,
            "prevalence": 0.016994842944210033,
            "ap": 0.07467278985000228,
            "auroc": 0.7814107492575401
          },
          "Hernia": {
            "positives": 86,
            "prevalence": 0.0033598999843725583,
            "ap": 0.03505541064192217,
            "auroc": 0.8713012680845633
          },
          "Infiltration": {
            "positives": 6112,
            "prevalence": 0.23878731051726831,
            "ap": 0.37916834755953005,
            "auroc": 0.6811102634152227
          },
          "Mass": {
            "positives": 1748,
            "prevalence": 0.06829192061259572,
            "ap": 0.19046255354120534,
            "auroc": 0.724831469827596
          },
          "Nodule": {
            "positives": 1623,
            "prevalence": 0.06340834505391467,
            "ap": 0.16932706672361175,
            "auroc": 0.6975860653874344
          },
          "Pleural_Thickening": {
            "positives": 1143,
            "prevalence": 0.044655414908579466,
            "ap": 0.10038062000329298,
            "auroc": 0.7225424752016822
          },
          "Pneumonia": {
            "positives": 555,
            "prevalence": 0.021683075480543833,
            "ap": 0.041157960327013514,
            "auroc": 0.6705020343213707
          },
          "Pneumothorax": {
            "positives": 2665,
            "prevalence": 0.10411783091107986,
            "ap": 0.35416998553487594,
            "auroc": 0.819602506025295
          }
        }
      },
      "calibrated_thresholds": {
        "thresholds": [
          -1.5961288176345312,
          -2.0958394387940102,
          -2.444626164394874,
          -1.9552137644231844,
          -1.3157366287200507,
          -2.4062744060939707,
          -1.7720832408401597,
          -3.2448679584699445,
          -1.4449203969741737,
          -1.8459227698259426,
          -1.8254896478172138,
          -2.519672233110607,
          -3.612101490761695,
          -1.5870430778509603
        ],
        "global_fallback_classes": [],
        "metrics": {
          "micro_f1": 0.3274304216198525,
          "macro_f1": 0.2514835044957849,
          "micro_precision": 0.2400548461736224,
          "micro_recall": 0.5148129089171506,
          "exact_match": 0.1749492108141897,
          "mean_predicted_labels": 2.279457727769964,
          "per_class_f1": [
            0.3352726555566512,
            0.24029000517866392,
            0.21079904653331952,
            0.21095717884130982,
            0.4509044948258156,
            0.3041700735895339,
            0.09722222222222222,
            0.050955414012738856,
            0.45285484519501407,
            0.2619877942458588,
            0.241109709962169,
            0.17830478700043917,
            0.07945516458569807,
            0.40648567119155354
          ]
        }
      },
      "fixed_probability_0_5": {
        "micro_f1": 0.15694572217111316,
        "macro_f1": 0.0799689129093638,
        "micro_precision": 0.4735381199111769,
        "micro_recall": 0.09406013379401602,
        "exact_match": 0.37165963431786214,
        "mean_predicted_labels": 0.2111267385528989,
        "per_class_f1": [
          0.08227498619547212,
          0.03967538322813345,
          0.0010934937124111536,
          0.012486992715920915,
          0.27820857730653586,
          0.17357910906298002,
          0.0,
          0.0,
          0.23734101271898247,
          0.07735946364105209,
          0.03523194362889019,
          0.0017241379310344827,
          0.0,
          0.18058968058968058
        ]
      },
      "verification": "Training, calibration and evaluation image IDs/patients are pairwise disjoint.",
      "zero_shot_same_encoder": {
        "ranking": {
          "num_images": 25596,
          "num_classes": 14,
          "mean_labels": 1.062900453195812,
          "macro_ap": 0.08632057640236498,
          "micro_ap": 0.08819675910328062,
          "macro_auroc": 0.5444838355236122,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 3279,
              "prevalence": 0.12810595405532113,
              "ap": 0.14088201591060245,
              "auroc": 0.5167774378232921
            },
            "Cardiomegaly": {
              "positives": 1069,
              "prevalence": 0.04176433817784029,
              "ap": 0.07063716348080068,
              "auroc": 0.5455345730557984
            },
            "Consolidation": {
              "positives": 1815,
              "prevalence": 0.07090951711204876,
              "ap": 0.05940714537900944,
              "auroc": 0.43929117661470835
            },
            "Edema": {
              "positives": 925,
              "prevalence": 0.03613845913423973,
              "ap": 0.04493150910969028,
              "auroc": 0.5570623787420836
            },
            "Effusion": {
              "positives": 4658,
              "prevalence": 0.18198155961869042,
              "ap": 0.17581225659884075,
              "auroc": 0.47567797743945495
            },
            "Emphysema": {
              "positives": 1093,
              "prevalence": 0.042701984685107046,
              "ap": 0.04792237847244621,
              "auroc": 0.5150278291819226
            },
            "Fibrosis": {
              "positives": 435,
              "prevalence": 0.016994842944210033,
              "ap": 0.03565197362990995,
              "auroc": 0.6509326831755221
            },
            "Hernia": {
              "positives": 86,
              "prevalence": 0.0033598999843725583,
              "ap": 0.004528655513151009,
              "auroc": 0.620796678001331
            },
            "Infiltration": {
              "positives": 6112,
              "prevalence": 0.23878731051726831,
              "ap": 0.20996781277757165,
              "auroc": 0.4520818439361173
            },
            "Mass": {
              "positives": 1748,
              "prevalence": 0.06829192061259572,
              "ap": 0.07465765210537752,
              "auroc": 0.5351915583593115
            },
            "Nodule": {
              "positives": 1623,
              "prevalence": 0.06340834505391467,
              "ap": 0.09426417346891844,
              "auroc": 0.5896482716397495
            },
            "Pleural_Thickening": {
              "positives": 1143,
              "prevalence": 0.044655414908579466,
              "ap": 0.0584593109584407,
              "auroc": 0.5581500662312929
            },
            "Pneumonia": {
              "positives": 555,
              "prevalence": 0.021683075480543833,
              "ap": 0.034475544707343195,
              "auroc": 0.6055116815629575
            },
            "Pneumothorax": {
              "positives": 2665,
              "prevalence": 0.10411783091107986,
              "ap": 0.15689047752100743,
              "auroc": 0.5610895415670292
            }
          }
        },
        "calibrated_metrics": {
          "micro_f1": 0.2178189164626134,
          "macro_f1": 0.13409526982418848,
          "micro_precision": 0.12926763911696199,
          "micro_recall": 0.6915386311842976,
          "exact_match": 0.0012501953430223472,
          "mean_predicted_labels": 5.686161900296922,
          "per_class_f1": [
            0.22672431829276798,
            0.0979020979020979,
            0.1331327290741624,
            0.08147005911076843,
            0.3077791427338225,
            0.04225352112676056,
            0.052303860523038606,
            0.0,
            0.38562215477996964,
            0.12806096027224975,
            0.13276001074979843,
            0.10309278350515463,
            0.06753246753246753,
            0.11869967193558008
          ]
        }
      }
    }
  },
  "verification": {
    "plant": {
      "num_evaluation": 1864,
      "max_saved_head_logit_difference": 4.263256414560601e-14,
      "selected_optimizer": {
        "iterations": 213,
        "function_evaluations": 252,
        "objective": 0.10268108299469493,
        "max_abs_gradient": 3.1384996772214975e-08,
        "max_iter": 1500,
        "hit_iteration_limit": false
      },
      "selected_reached_default_function_eval_limit": false,
      "all_candidates_finished_before_iteration_limit": true
    },
    "bigearth": {
      "num_evaluation": 119825,
      "max_saved_head_logit_difference": 5.684341886080802e-14,
      "selected_optimizer": {
        "iterations": 1273,
        "function_evaluations": 1876,
        "objective": 0.16439414580623884,
        "max_abs_gradient": 1.3639702568906743e-07,
        "max_iter": 1500,
        "hit_iteration_limit": false
      },
      "selected_reached_default_function_eval_limit": true,
      "all_candidates_finished_before_iteration_limit": false
    },
    "nih": {
      "num_evaluation": 25596,
      "max_saved_head_logit_difference": 2.1316282072803006e-14,
      "selected_optimizer": {
        "iterations": 1160,
        "function_evaluations": 1318,
        "objective": 0.13940538418117715,
        "max_abs_gradient": 7.037429574746997e-08,
        "max_iter": 1500,
        "hit_iteration_limit": false
      },
      "selected_reached_default_function_eval_limit": false,
      "all_candidates_finished_before_iteration_limit": true
    },
    "nih-pecore": {
      "num_evaluation": 25596,
      "max_saved_head_logit_difference": 2.842170943040401e-14,
      "selected_optimizer": {
        "iterations": 598,
        "function_evaluations": 672,
        "objective": 0.13611297308649903,
        "max_abs_gradient": 6.606753893914182e-08,
        "max_iter": 1500,
        "hit_iteration_limit": false
      },
      "selected_reached_default_function_eval_limit": false,
      "all_candidates_finished_before_iteration_limit": true
    }
  }
}

nih-biomedclip.md

BiomedCLIP on NIH ChestX-ray14

Microsoft BiomedCLIP loads and runs in this checkout of OpenCLIP without compatibility fixes. Its disease ranking improves over the generic encoders tested here, but the resulting multilabel predictions remain weak.

Full held-out test results

All 25,596 test images from 2,797 patients; 14 labels. Percentages below are macro average precision (mAP), macro AUROC, and F1, not classification accuracy.

Model / prompts	mAP	AUROC	Macro F1	Micro F1
PE-Core-L, original prompts	8.63	54.44	13.37	21.78
SigLIP2-L, original prompts	9.70	56.86	14.41	23.75
BiomedCLIP, identical original prompts	12.28	62.96	17.67	22.19
BiomedCLIP, calibration-selected descriptions + pairs	12.92	64.33	18.44	22.00

The first three rows use the same two positive prompts per label with cosine scoring. The last row combines the original prompts with three visual descriptions/synonyms and uses positive-minus-negative similarity. Prompt selection maximized macro AP on a separate training calibration fold before loading test images. Model weights remain frozen. F1 uses per-class thresholds fitted on that calibration fold.

For the selected variant, micro precision is 13.75% and recall 55.02%. It predicts 4.25 labels per image on average versus 1.06 annotated labels. Better ranking does not make these thresholded outputs reliable. The selected prompts improved test mAP by only 0.63 percentage point over BiomedCLIP's original prompts; the stronger development improvement did not fully carry over.

Calibration-only prompt comparison

All 8,668 images from training patient fold 0, with 2,797 patients. No image or patient overlaps the test split. Six unchanged candidate families from the earlier prompt experiment were evaluated; the last row won by calibration mAP.

BiomedCLIP prompt family	Development mAP	Development AUROC
Original prompts	8.17	59.96
Original positive/negative pairs	9.55	67.83
Six domain templates	8.42	61.04
Visual descriptions / synonyms	8.66	60.08
Original + descriptions	8.76	60.74
Original + descriptions, paired	9.89	67.16

Development and test label prevalences differ; compare prompt strategies within each split. The highest-AUROC candidate was not substituted after observing test results. The original baseline and the single mAP-selected family were the only BiomedCLIP prompt families scored on test images.

Model and reproducibility

• Model: hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224.
• Cached model repository revision: 9f341de24bfb00180f1b847274256e9b65a3a32e.
• Dataset: timm/nih-chest-xray-14, revision c1bf579641b3256b4d49924436984b82bee5834d.
• Native model preprocessing: 224-pixel image input; PubMedBERT text encoder with 256-token context. All candidate prompts fit the context.
• CUDA image inference in bfloat16; float32 normalized embeddings, text encoding, and scoring. The same image embeddings are reused for the two test prompt choices.
• Verified complete counts, unique IDs, identical image/label populations to both earlier generic model runs, and patient/image disjointness between calibration and test. All 14 classes have positive and negative examples; no threshold fallback was needed.
• No confidence intervals or audit of pretraining overlap were performed. Results measure agreement with this dataset's labels, not clinical validation.

BiomedCLIP was trained on 15 million figure-caption pairs from biomedical papers, covering multiple biomedical image types (official model card). This experiment tests that general biomedical checkpoint on chest X-ray labels.

The baseline comparison is in the full comparison report.

prompt-comparison-summary.json

{
  "reports": {
    "bigearth-pecore": {
      "protocol": "Exploratory calibration-only prompt comparison; no held-out evaluation",
      "data": {
        "dataset": "bigearthnet-v2-rgb",
        "revision": "edb997737d3888abe5257c71d8e4fabc3c8ec34d",
        "split": "validation",
        "fold": null,
        "eligible_images": 122342,
        "selected_images": 8192,
        "sampling": "lowest SHA256(42:image_id), no label stratification",
        "prior_calibration_matching_ids": 8192
      },
      "model": "PE-Core-L-14-336",
      "pretrained": "meta",
      "context_length": 32,
      "amp": "bfloat16 image; float32 text/scoring",
      "classnames": [
        "agriculture_with_natural_vegetation",
        "agro_forestry_areas",
        "arable_land",
        "beaches_dunes_sands",
        "broad_leaved_forest",
        "coastal_wetlands",
        "complex_cultivation_patterns",
        "coniferous_forest",
        "industrial_commercial_units",
        "inland_waters",
        "inland_wetlands",
        "marine_waters",
        "mixed_forest",
        "moors_heathland_sclerophyllous_vegetation",
        "natural_grassland_sparse_vegetation",
        "pastures",
        "permanent_crops",
        "transitional_woodland_shrub",
        "urban_fabric"
      ],
      "results": {
        "baseline": {
          "score": "cosine",
          "max_prompt_tokens": 16,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.38183228380356815,
          "micro_ap": 0.3658420620977521,
          "macro_auroc": 0.7427822827377193,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.2879754427990232,
              "auroc": 0.5806295723891896
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.0847584525378261,
              "auroc": 0.6142768616556398
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.7710162072148576,
              "auroc": 0.8364094265410871
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.6465690765722656,
              "auroc": 0.9788906575241827
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.3866608285878585,
              "auroc": 0.6459407542982947
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.35446669073920056,
              "auroc": 0.8928658129100009
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.2395616536583628,
              "auroc": 0.6028948426490512
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.6405372595527927,
              "auroc": 0.7998638326836883
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.223591053870654,
              "auroc": 0.8530622908800027
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.6427939384271352,
              "auroc": 0.8949520582310332
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.14532071584334372,
              "auroc": 0.7399806309829776
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.4105971745575413,
              "auroc": 0.64391271286349
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.6191116532574912,
              "auroc": 0.7757503842187387
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.07965482823712136,
              "auroc": 0.6919860164183975
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.03873349399041231,
              "auroc": 0.6437714060515333
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.5580429909964433,
              "auroc": 0.7917978314492168
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.07511326250180345,
              "auroc": 0.5599164908159955
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.44923087013921337,
              "auroc": 0.7036052754448303
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.6010777987844483,
              "auroc": 0.8623565140093195
            }
          }
        },
        "baseline_paired": {
          "score": "paired",
          "max_prompt_tokens": 18,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.2573837344971687,
          "micro_ap": 0.2539110906196979,
          "macro_auroc": 0.6363536897469103,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.30222855644655966,
              "auroc": 0.613026687161972
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.05413261073802638,
              "auroc": 0.41665980969701033
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.6224478762174228,
              "auroc": 0.7070292034975558
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.01831798751408381,
              "auroc": 0.8335569976735644
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.25764042675377885,
              "auroc": 0.472578751106607
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.022318543132708524,
              "auroc": 0.7942882860766745
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.16016973736665047,
              "auroc": 0.3516509939817456
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.38108661994296095,
              "auroc": 0.6447427663872899
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.04533157704949982,
              "auroc": 0.7157243902369095
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.38749508040996833,
              "auroc": 0.8423707286695057
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.061309916846518486,
              "auroc": 0.6559833500949827
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.8893072038926481,
              "auroc": 0.9785627185716914
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.433378631693398,
              "auroc": 0.6599040972809758
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.03460921447907827,
              "auroc": 0.557666885526524
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.02771064189160039,
              "auroc": 0.5554129666802757
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.6595480277117729,
              "auroc": 0.7934904678892205
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.04528514716817587,
              "auroc": 0.29902003164505164
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.31278965048325735,
              "auroc": 0.5623186226684499
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.17518350570809652,
              "auroc": 0.6367323503452902
            }
          }
        },
        "domain_templates": {
          "score": "cosine",
          "max_prompt_tokens": 19,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.38110640473993185,
          "micro_ap": 0.36018516255784927,
          "macro_auroc": 0.7358596188734904,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.30045262874676704,
              "auroc": 0.5952537622884528
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.07357737289799202,
              "auroc": 0.5654910334080858
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.7671957923812688,
              "auroc": 0.8353086544218731
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.6572871306298922,
              "auroc": 0.9870846087914779
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.40408394408838577,
              "auroc": 0.6529889674191593
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.37967422055978634,
              "auroc": 0.8984002291231946
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.24228598456083195,
              "auroc": 0.6044754130736703
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.633003949653097,
              "auroc": 0.7987957904633288
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.23670224844247933,
              "auroc": 0.8682755466866218
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.6135453906408483,
              "auroc": 0.881873145297723
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.14388595624477812,
              "auroc": 0.7369802957499907
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.3390200565950458,
              "auroc": 0.519423611450885
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.6063127304218313,
              "auroc": 0.7711513687474613
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.07385214356290977,
              "auroc": 0.6662138026313988
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.04338450007521952,
              "auroc": 0.6565250181455271
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.5613547235225416,
              "auroc": 0.7911725825620872
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.07760209082261407,
              "auroc": 0.5775839789260464
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.43480572664063366,
              "auroc": 0.6908932469693664
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.6529950995717825,
              "auroc": 0.8834417024399657
            }
          }
        },
        "visual_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 19,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.4062451243212029,
          "micro_ap": 0.3258596846242155,
          "macro_auroc": 0.7670991411882425,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.30341043685062224,
              "auroc": 0.5954454035909431
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.10659808183151524,
              "auroc": 0.642533155191586
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.7723478649630919,
              "auroc": 0.8327352314535794
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.694822586851569,
              "auroc": 0.9942598261295458
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.40894420144234966,
              "auroc": 0.661389643614295
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.39102142285154684,
              "auroc": 0.8939977905977661
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.26599899408270067,
              "auroc": 0.6517521918989284
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.6504737030456706,
              "auroc": 0.8134034288757586
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.3370192009306242,
              "auroc": 0.9070894366984629
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.6637242815188783,
              "auroc": 0.9003751635667769
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.2026100948002244,
              "auroc": 0.8062506052817819
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.3840847260741126,
              "auroc": 0.6189647419543484
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.6044370330711009,
              "auroc": 0.7701726577657839
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.06758111803508964,
              "auroc": 0.6797403677236425
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.039653520189133734,
              "auroc": 0.6590462540827435
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.543687368181697,
              "auroc": 0.7906653128596768
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.16897707998918923,
              "auroc": 0.7527085383771756
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.47728389746277033,
              "auroc": 0.7224022766917906
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.6359817499309683,
              "auroc": 0.8819516562220233
            }
          }
        },
        "baseline_plus_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 19,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.40086300981190387,
          "micro_ap": 0.33352269841617915,
          "macro_auroc": 0.7606395376180519,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.296865422655178,
              "auroc": 0.5898601856753515
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.094401788578485,
              "auroc": 0.6305257877844597
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.7742002817034939,
              "auroc": 0.8354692356211815
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.6937878868847485,
              "auroc": 0.9906011999510224
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.4050454655619149,
              "auroc": 0.6576826556821359
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.37958063081100024,
              "auroc": 0.8941614500225032
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.25880197342813055,
              "auroc": 0.638516385621313
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.6477673778652918,
              "auroc": 0.8088982233374954
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.29929567899788906,
              "auroc": 0.894769097117692
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.6688569963211023,
              "auroc": 0.9016368023356957
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.184012717819877,
              "auroc": 0.7878675457220546
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.3964164806124553,
              "auroc": 0.6280714346457537
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.6130866818816543,
              "auroc": 0.7743629965070076
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.07517194500774867,
              "auroc": 0.6865760743185544
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.03933774540461581,
              "auroc": 0.6537464843041191
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.5538432283383534,
              "auroc": 0.7930767928915525
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.12722607416143064,
              "auroc": 0.6880682293687566
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.46660149497961023,
              "auroc": 0.716485750962918
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.6420973154131937,
              "auroc": 0.881774882873421
            }
          }
        },
        "descriptions_paired": {
          "score": "paired",
          "max_prompt_tokens": 19,
          "num_images": 8192,
          "num_classes": 19,
          "mean_labels": 3.0133056640625,
          "macro_ap": 0.3164423766721027,
          "micro_ap": 0.22141082374976429,
          "macro_auroc": 0.7138939601474509,
          "ap_classes": 19,
          "auroc_classes": 19,
          "per_class": {
            "agriculture_with_natural_vegetation": {
              "positives": 2138,
              "prevalence": 0.260986328125,
              "ap": 0.31840651912501444,
              "auroc": 0.6355874769729126
            },
            "agro_forestry_areas": {
              "positives": 564,
              "prevalence": 0.06884765625,
              "ap": 0.06823447928862075,
              "auroc": 0.5111336267651467
            },
            "arable_land": {
              "positives": 3368,
              "prevalence": 0.4111328125,
              "ap": 0.6938307866002752,
              "auroc": 0.7607006535020857
            },
            "beaches_dunes_sands": {
              "positives": 25,
              "prevalence": 0.0030517578125,
              "ap": 0.05127814452392225,
              "auroc": 0.9042096240969756
            },
            "broad_leaved_forest": {
              "positives": 2352,
              "prevalence": 0.287109375,
              "ap": 0.32007089227714663,
              "auroc": 0.5396901354719971
            },
            "coastal_wetlands": {
              "positives": 45,
              "prevalence": 0.0054931640625,
              "ap": 0.039060717606590814,
              "auroc": 0.8006409994135537
            },
            "complex_cultivation_patterns": {
              "positives": 1756,
              "prevalence": 0.21435546875,
              "ap": 0.2460533507430258,
              "auroc": 0.6124108711532935
            },
            "coniferous_forest": {
              "positives": 2662,
              "prevalence": 0.324951171875,
              "ap": 0.4093258782872762,
              "auroc": 0.6670127628413014
            },
            "industrial_commercial_units": {
              "positives": 174,
              "prevalence": 0.021240234375,
              "ap": 0.24498294144784427,
              "auroc": 0.8220498132076391
            },
            "inland_waters": {
              "positives": 1122,
              "prevalence": 0.136962890625,
              "ap": 0.5244336886105502,
              "auroc": 0.8708488075698326
            },
            "inland_wetlands": {
              "positives": 342,
              "prevalence": 0.041748046875,
              "ap": 0.08685190492540423,
              "auroc": 0.7343733005549968
            },
            "marine_waters": {
              "positives": 948,
              "prevalence": 0.11572265625,
              "ap": 0.8643412826456264,
              "auroc": 0.975761040127491
            },
            "mixed_forest": {
              "positives": 2894,
              "prevalence": 0.353271484375,
              "ap": 0.4527108771940832,
              "auroc": 0.6503281088454967
            },
            "moors_heathland_sclerophyllous_vegetation": {
              "positives": 258,
              "prevalence": 0.031494140625,
              "ap": 0.03359001336871351,
              "auroc": 0.529993815254923
            },
            "natural_grassland_sparse_vegetation": {
              "positives": 176,
              "prevalence": 0.021484375,
              "ap": 0.030584517812777278,
              "auroc": 0.5962981707040464
            },
            "pastures": {
              "positives": 1699,
              "prevalence": 0.2073974609375,
              "ap": 0.657229010326583,
              "auroc": 0.7960638463643601
            },
            "permanent_crops": {
              "positives": 569,
              "prevalence": 0.0694580078125,
              "ap": 0.15758242049075574,
              "auroc": 0.7281168796586595
            },
            "transitional_woodland_shrub": {
              "positives": 2476,
              "prevalence": 0.30224609375,
              "ap": 0.4197397915645644,
              "auroc": 0.6699496411173578
            },
            "urban_fabric": {
              "positives": 1117,
              "prevalence": 0.1363525390625,
              "ap": 0.39409793993117725,
              "auroc": 0.758815669179497
            }
          }
        }
      }
    },
    "nih-biomedclip": {
      "protocol": "Exploratory calibration-only prompt comparison; no held-out evaluation",
      "data": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "split": "train",
        "fold": 0,
        "eligible_images": 8668,
        "selected_images": 8668,
        "sampling": "full fold"
      },
      "model": "hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
      "pretrained": null,
      "context_length": 256,
      "amp": "bfloat16 image; float32 text/scoring",
      "classnames": [
        "Atelectasis",
        "Cardiomegaly",
        "Consolidation",
        "Edema",
        "Effusion",
        "Emphysema",
        "Fibrosis",
        "Hernia",
        "Infiltration",
        "Mass",
        "Nodule",
        "Pleural_Thickening",
        "Pneumonia",
        "Pneumothorax"
      ],
      "results": {
        "baseline": {
          "score": "cosine",
          "max_prompt_tokens": 13,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.08165282865662966,
          "micro_ap": 0.04786582488176502,
          "macro_auroc": 0.5996130327123027,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.08858326897774321,
              "auroc": 0.46467468327910877
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.18303867327645523,
              "auroc": 0.8407493804770672
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.057124662846577356,
              "auroc": 0.6431501932691684
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.0411299364258515,
              "auroc": 0.7141164854326899
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.32971112240018213,
              "auroc": 0.7807414365831464
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.018738423781740694,
              "auroc": 0.41840699368625545
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.04624201187961471,
              "auroc": 0.6337413086737681
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.004977812239190723,
              "auroc": 0.6779276304929182
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.15922685770043193,
              "auroc": 0.49083351749319604
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.056840048612792196,
              "auroc": 0.4972862334667849
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.051827915520633996,
              "auroc": 0.477796754910333
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04875404039999905,
              "auroc": 0.5971381306252961
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.01661636604992719,
              "auroc": 0.6061360987497352
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.04032846108167527,
              "auroc": 0.5518836108327696
            }
          }
        },
        "baseline_paired": {
          "score": "paired",
          "max_prompt_tokens": 14,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.09546748649208835,
          "micro_ap": 0.0917810384736494,
          "macro_auroc": 0.6783200938561595,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10993758814138752,
              "auroc": 0.5452144490042394
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.05423864395520831,
              "auroc": 0.7680153115611977
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.0811303848619892,
              "auroc": 0.7497632007969344
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.06430922368082352,
              "auroc": 0.8345552972542388
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.3068111355110303,
              "auroc": 0.8141275731395928
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.02928302173691847,
              "auroc": 0.67139578026451
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.022110811742728005,
              "auroc": 0.6035612782394943
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0017400292050710722,
              "auroc": 0.45933342137409616
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.2373390009240637,
              "auroc": 0.6213970367184902
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.18644520930365843,
              "auroc": 0.7520114567242955
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.1011940582399577,
              "auroc": 0.5975899473345719
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.051001378847753936,
              "auroc": 0.6612709996954728
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.02329209459817038,
              "auroc": 0.6957863159567704
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.06771223014047639,
              "auroc": 0.7224592459223304
            }
          }
        },
        "domain_templates": {
          "score": "cosine",
          "max_prompt_tokens": 15,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.08420250910146279,
          "micro_ap": 0.049405150842518814,
          "macro_auroc": 0.6103763034061335,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10459900479403242,
              "auroc": 0.5180824367790594
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.18200045667365564,
              "auroc": 0.8388285132834024
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.057127966981428545,
              "auroc": 0.6456542166581909
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.042137105833933444,
              "auroc": 0.7309956731439738
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.3331265880595745,
              "auroc": 0.780190562258863
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.0190167849322831,
              "auroc": 0.43527585876506986
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.04701677156798067,
              "auroc": 0.6490680088961722
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.005344047726674798,
              "auroc": 0.6896480570504144
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.16203433189211397,
              "auroc": 0.49329832305024474
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.06608795311346993,
              "auroc": 0.5195144102233851
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.05348651040248885,
              "auroc": 0.4835617712140038
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04823651623472291,
              "auroc": 0.5938488656357854
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.014518369720511962,
              "auroc": 0.5920229391820301
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.044102719487608216,
              "auroc": 0.5752786115452734
            }
          }
        },
        "visual_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 18,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.08655540959375159,
          "micro_ap": 0.051155695309059064,
          "macro_auroc": 0.6008018708712514,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.12020747718448084,
              "auroc": 0.5521966349945775
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.15797296607135455,
              "auroc": 0.826806307366439
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.09639465860356101,
              "auroc": 0.7298413037245386
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.08587021295046422,
              "auroc": 0.8446601360260176
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.2531007149157007,
              "auroc": 0.6546993339186434
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.01185809590755956,
              "auroc": 0.3424611709666868
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.034754120279906583,
              "auroc": 0.5583406297553553
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0029608949725595485,
              "auroc": 0.6669252864076067
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.20644347797059892,
              "auroc": 0.5572783959576413
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.09501496112608232,
              "auroc": 0.5577518280170349
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.051525521652686035,
              "auroc": 0.47210944297108337
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04370517972488671,
              "auroc": 0.5558046288150504
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.017718237069385363,
              "auroc": 0.5971710108073746
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.03424921588329583,
              "auroc": 0.49518008246946976
            }
          }
        },
        "baseline_plus_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 18,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.08755780567658747,
          "micro_ap": 0.0511126329884568,
          "macro_auroc": 0.6074150410202623,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10576896729517521,
              "auroc": 0.5151389042936015
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.16995088604343508,
              "auroc": 0.8350269267509309
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.08326449042309685,
              "auroc": 0.723982328480153
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.0756650573529741,
              "auroc": 0.8240525964986434
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.30776516295653095,
              "auroc": 0.7264522687082663
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.0156185221559238,
              "auroc": 0.3658717493796936
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.04145619143468996,
              "auroc": 0.5828800187287838
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0032781955596327293,
              "auroc": 0.6855789230413681
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.18875020023351685,
              "auroc": 0.5399433783081582
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.07662217753352918,
              "auroc": 0.5336497743031319
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.051636589071561774,
              "auroc": 0.4738562370832683
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.05198638950450711,
              "auroc": 0.5772373240508899
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.01728730157299556,
              "auroc": 0.6037342392456029
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.036759148334655406,
              "auroc": 0.5164059054111825
            }
          }
        },
        "descriptions_paired": {
          "score": "paired",
          "max_prompt_tokens": 18,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.09894041589114187,
          "micro_ap": 0.09820786794667975,
          "macro_auroc": 0.6716299157086727,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.11745595840579064,
              "auroc": 0.5773297317115252
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.04598343780274545,
              "auroc": 0.7445329696952917
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.09142105679147514,
              "auroc": 0.755201315946433
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.06291103974773576,
              "auroc": 0.8360098464423866
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.3311905259264018,
              "auroc": 0.8072416440860488
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.019971905393359947,
              "auroc": 0.5458935881297637
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.025139251824974593,
              "auroc": 0.5630818213742245
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.001781024270441182,
              "auroc": 0.5195698108224107
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.2571743975530675,
              "auroc": 0.6224501324955553
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.18476784975431398,
              "auroc": 0.7482823169843836
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.09828040011033544,
              "auroc": 0.5954279529018307
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.05773974001134218,
              "auroc": 0.677331114231576
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.02389770101831671,
              "auroc": 0.6965750158931977
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.06745153386568571,
              "auroc": 0.7138915592067911
            }
          }
        }
      }
    },
    "nih-pecore": {
      "protocol": "Exploratory calibration-only prompt comparison; no held-out evaluation",
      "data": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "split": "train",
        "fold": 0,
        "eligible_images": 8668,
        "selected_images": 8668,
        "sampling": "full fold",
        "prior_calibration_matching_ids": 8668
      },
      "model": "PE-Core-L-14-336",
      "pretrained": "meta",
      "context_length": 32,
      "amp": "bfloat16 image; float32 text/scoring",
      "classnames": [
        "Atelectasis",
        "Cardiomegaly",
        "Consolidation",
        "Edema",
        "Effusion",
        "Emphysema",
        "Fibrosis",
        "Hernia",
        "Infiltration",
        "Mass",
        "Nodule",
        "Pleural_Thickening",
        "Pneumonia",
        "Pneumothorax"
      ],
      "results": {
        "baseline": {
          "score": "cosine",
          "max_prompt_tokens": 13,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.05216358615202941,
          "micro_ap": 0.048522462099207575,
          "macro_auroc": 0.5362613786514622,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.11188698031164322,
              "auroc": 0.5174778788326925
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.04419731875330831,
              "auroc": 0.572743067572993
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.02606837901084473,
              "auroc": 0.40130129690204275
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.023163032288270855,
              "auroc": 0.5889405491522117
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.10455856150655919,
              "auroc": 0.4911409433123384
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.02251541666304295,
              "auroc": 0.5168316962530519
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.03599190382264414,
              "auroc": 0.6280060868547349
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.00329133189241564,
              "auroc": 0.6353750536498398
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.14668242430252615,
              "auroc": 0.4613476321023491
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.04525122849609274,
              "auroc": 0.48471010674628723
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.0697204087952132,
              "auroc": 0.5446393442580489
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.03806228569591706,
              "auroc": 0.5690713926202883
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.015388057654781406,
              "auroc": 0.575495337995338
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.043512876935152156,
              "auroc": 0.5205789148682547
            }
          }
        },
        "baseline_paired": {
          "score": "paired",
          "max_prompt_tokens": 14,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.06741308710606161,
          "micro_ap": 0.061985870024804726,
          "macro_auroc": 0.611831842944416,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10996540455987094,
              "auroc": 0.5389334085083308
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.02613547185483694,
              "auroc": 0.5523714922542329
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.040187399361604606,
              "auroc": 0.5236303211805011
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.05130673109231883,
              "auroc": 0.7653435687963264
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.16099366204014443,
              "auroc": 0.6890406942496535
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.027135391441041094,
              "auroc": 0.5976367234606325
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.052729891388102024,
              "auroc": 0.6783017675289712
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.003434042479313543,
              "auroc": 0.6386848360791046
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.16879521572165504,
              "auroc": 0.5119987118764198
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.07788620698330412,
              "auroc": 0.5897561993458018
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.09269559533527627,
              "auroc": 0.6035059691197675
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.05983212176971037,
              "auroc": 0.64470889092847
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.017177868652765032,
              "auroc": 0.5900786713286713
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.05550821680491925,
              "auroc": 0.6416545465649408
            }
          }
        },
        "domain_templates": {
          "score": "cosine",
          "max_prompt_tokens": 15,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.04956133693662019,
          "micro_ap": 0.04589084930001851,
          "macro_auroc": 0.5233827944651583,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.11367374685935627,
              "auroc": 0.5196779028640441
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.04439959381034707,
              "auroc": 0.5724132321791979
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.025776548508201807,
              "auroc": 0.3965217828060549
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.01902442798199351,
              "auroc": 0.5387650192898891
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.09945548860470879,
              "auroc": 0.4694802747918606
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.021646310171905644,
              "auroc": 0.5175725948465836
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.03065691882210376,
              "auroc": 0.6234610792461666
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.003671753972887727,
              "auroc": 0.649014493710588
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.1421901172994352,
              "auroc": 0.44497930441326666
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.03995651788708815,
              "auroc": 0.44575138968324385
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.06290730132781495,
              "auroc": 0.5255767800710063
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.034011014205730244,
              "auroc": 0.5561905748798809
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.014654682504110554,
              "auroc": 0.5588889330366603
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.04183429515699891,
              "auroc": 0.509065760693775
            }
          }
        },
        "visual_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 21,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.053406557481985355,
          "micro_ap": 0.03990036992012098,
          "macro_auroc": 0.5236075755009189,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09866492882630876,
              "auroc": 0.49174538474810214
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.051774707472158045,
              "auroc": 0.5951760275774232
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.027316569980402616,
              "auroc": 0.4275823460595901
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.025713884883717793,
              "auroc": 0.5220667946099541
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.09122791963537863,
              "auroc": 0.4226780099613233
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.014340835599283468,
              "auroc": 0.448120991961622
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.04868458395318514,
              "auroc": 0.6353276366615943
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.002328603290766663,
              "auroc": 0.5907177523193239
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.16711270146581259,
              "auroc": 0.5059872362283264
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.06692702889263971,
              "auroc": 0.535005606769555
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.07179885040023629,
              "auroc": 0.551088806504995
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.030975727435683706,
              "auroc": 0.5482387642112743
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.01379789822229741,
              "auroc": 0.5538964823055732
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.03702756468992425,
              "auroc": 0.5028742170942065
            }
          }
        },
        "baseline_plus_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 21,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.05302059942617811,
          "micro_ap": 0.04465679892141195,
          "macro_auroc": 0.5303719698707781,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10391008506426264,
              "auroc": 0.5024609798629597
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.05109131405326479,
              "auroc": 0.5867048302022622
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.026377930556653495,
              "auroc": 0.4121270909589374
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.024637612557520294,
              "auroc": 0.5549986438467864
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.09631056946545652,
              "auroc": 0.44290680485195666
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.01745623032598919,
              "auroc": 0.48038435869734003
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.042067254003761254,
              "auroc": 0.633832611494791
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.002615399152569331,
              "auroc": 0.6161395225989634
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.16050521517232663,
              "auroc": 0.49163212424917524
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.05745036125798335,
              "auroc": 0.5219639755673946
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.07231220846007635,
              "auroc": 0.5502574993129639
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.033229215620247776,
              "auroc": 0.5576553194998985
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.014387918699207136,
              "auroc": 0.564233682983683
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.03993707757717504,
              "auroc": 0.5099101340637815
            }
          }
        },
        "descriptions_paired": {
          "score": "paired",
          "max_prompt_tokens": 21,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.058612209478017915,
          "micro_ap": 0.039707750313166146,
          "macro_auroc": 0.5641332354455754,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09036440604653334,
              "auroc": 0.4757932656758356
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.030707597212865305,
              "auroc": 0.5898173272603935
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.040995435425286594,
              "auroc": 0.5405664345762413
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.03065337003738463,
              "auroc": 0.7092531574412598
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.10807044963311632,
              "auroc": 0.5094435281295198
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.015743759872905466,
              "auroc": 0.46357991958318057
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.04032457516564407,
              "auroc": 0.6559812712162004
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0018996827382884139,
              "auroc": 0.5333248043844301
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.18264208846060495,
              "auroc": 0.5400866248175822
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.1015160636683236,
              "auroc": 0.5930821320435512
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.08216496675348933,
              "auroc": 0.578785361637407
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04012509244147747,
              "auroc": 0.5757175953339649
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.012054114264485057,
              "auroc": 0.5400409249841067
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.04330933097184625,
              "auroc": 0.5923929491543827
            }
          }
        }
      }
    },
    "nih-siglip2": {
      "protocol": "Exploratory calibration-only prompt comparison; no held-out evaluation",
      "data": {
        "dataset": "nih-chest-xray-14",
        "revision": "c1bf579641b3256b4d49924436984b82bee5834d",
        "split": "train",
        "fold": 0,
        "eligible_images": 8668,
        "selected_images": 8668,
        "sampling": "full fold",
        "prior_calibration_matching_ids": 8668
      },
      "model": "ViT-L-16-SigLIP2-256",
      "pretrained": "webli",
      "context_length": 64,
      "amp": "bfloat16 image; float32 text/scoring",
      "classnames": [
        "Atelectasis",
        "Cardiomegaly",
        "Consolidation",
        "Edema",
        "Effusion",
        "Emphysema",
        "Fibrosis",
        "Hernia",
        "Infiltration",
        "Mass",
        "Nodule",
        "Pleural_Thickening",
        "Pneumonia",
        "Pneumothorax"
      ],
      "results": {
        "baseline": {
          "score": "cosine",
          "max_prompt_tokens": 10,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.0585872492622929,
          "micro_ap": 0.056606037226372086,
          "macro_auroc": 0.5491378670767751,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09492638662103271,
              "auroc": 0.4780623798432416
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.05641149913756191,
              "auroc": 0.6421580164190882
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.05197205339788577,
              "auroc": 0.5876029391144568
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.03191137332957586,
              "auroc": 0.6616932492660944
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.11790385966301098,
              "auroc": 0.5143335367907679
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.018756801592853686,
              "auroc": 0.5203614957396266
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.02687987704094058,
              "auroc": 0.5679288306215615
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.002249248221791585,
              "auroc": 0.5302420020469477
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.20566464222615433,
              "auroc": 0.5524733665020177
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.07688640015157171,
              "auroc": 0.5744840195809109
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.06506613718278675,
              "auroc": 0.5415982544491577
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.028002528846228357,
              "auroc": 0.5075925636123706
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.013523287885999585,
              "auroc": 0.5150223829201103
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.030067394374706725,
              "auroc": 0.4943771021684992
            }
          }
        },
        "baseline_paired": {
          "score": "paired",
          "max_prompt_tokens": 11,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.0727137818138834,
          "micro_ap": 0.06874767602179997,
          "macro_auroc": 0.6239347889098977,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.10958644248560881,
              "auroc": 0.5268582704821059
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.06319396415417215,
              "auroc": 0.6918960170559163
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.059319492551892145,
              "auroc": 0.6681343403839433
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.027245267489184457,
              "auroc": 0.6960137651690228
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.18404829449807583,
              "auroc": 0.7017705976971618
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.020386502796568437,
              "auroc": 0.5392597787050711
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.03518737188080269,
              "auroc": 0.6240093643918997
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0026758241777994395,
              "auroc": 0.619399782099112
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.23441466584053386,
              "auroc": 0.6170452893898037
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.09292014299636095,
              "auroc": 0.651136440399364
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.08253450962072693,
              "auroc": 0.5898134240505887
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04173840991390744,
              "auroc": 0.6217067698111931
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.015195948953788726,
              "auroc": 0.5673321943208307
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.04954610803494569,
              "auroc": 0.6207110107825553
            }
          }
        },
        "domain_templates": {
          "score": "cosine",
          "max_prompt_tokens": 12,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.057107144468252984,
          "micro_ap": 0.05497123292184844,
          "macro_auroc": 0.5397673970398721,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09035392824698028,
              "auroc": 0.4693887564083605
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.05775673004472172,
              "auroc": 0.6310419060540197
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.0485570077674502,
              "auroc": 0.5710121360899565
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.026632969674393523,
              "auroc": 0.6046809104108931
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.11315314582720556,
              "auroc": 0.4971193801790623
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.0174563764769594,
              "auroc": 0.5067742249886842
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.028629886849472378,
              "auroc": 0.5778925436029498
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0025244908908894897,
              "auroc": 0.5326025950014857
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.2015914637294212,
              "auroc": 0.5495141663730063
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.07459072442234034,
              "auroc": 0.5683447945610582
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.06532828594659351,
              "auroc": 0.5382468652116784
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.0285839749963289,
              "auroc": 0.5130069110780267
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.013079532999259277,
              "auroc": 0.5059864378046197
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.0312615046835259,
              "auroc": 0.49113193079440765
            }
          }
        },
        "visual_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 17,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.058337972740055415,
          "micro_ap": 0.060901300198433525,
          "macro_auroc": 0.5434189336888963,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09873799541705809,
              "auroc": 0.5008139079907326
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.06105534983518954,
              "auroc": 0.6412387689837056
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.04743675106423686,
              "auroc": 0.5726373969039262
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.0441989792530888,
              "auroc": 0.7376215725045712
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.09994216575543705,
              "auroc": 0.44921137056703053
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.012461439675433849,
              "auroc": 0.3869786865693339
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.02537341353416293,
              "auroc": 0.5641278239494323
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.005163999086526415,
              "auroc": 0.5592871999735878
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.2145361626310729,
              "auroc": 0.5582294074432439
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.06816165832956093,
              "auroc": 0.5526974791303578
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.06630493911342958,
              "auroc": 0.5395320953992023
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.028398028868253912,
              "auroc": 0.4992838850071057
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.014498661196224333,
              "auroc": 0.5570936639118458
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.030462074601100574,
              "auroc": 0.4891118133104729
            }
          }
        },
        "baseline_plus_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 17,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.05910819878043737,
          "micro_ap": 0.05989632786706001,
          "macro_auroc": 0.5481602522213358,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.09701976798501562,
              "auroc": 0.49034578958394953
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.06129365298262862,
              "auroc": 0.6424113632273337
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.04922107357480851,
              "auroc": 0.5792912138392026
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.044179371862374174,
              "auroc": 0.7295111773548938
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.10608481427004388,
              "auroc": 0.47291391500846885
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.014731541619276842,
              "auroc": 0.441045286497309
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.02624197640593874,
              "auroc": 0.5665700573569004
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.0038781159005419168,
              "auroc": 0.5548012479778137
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.21304761820602042,
              "auroc": 0.5568756831335447
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.07366673901388078,
              "auroc": 0.5628298949650159
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.06599192769949735,
              "auroc": 0.5409485164553237
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.028300572020586702,
              "auroc": 0.5031459891216079
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.014126053577036245,
              "auroc": 0.5418865225683407
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.029731557808473377,
              "auroc": 0.4916668740089979
            }
          }
        },
        "descriptions_paired": {
          "score": "paired",
          "max_prompt_tokens": 17,
          "num_images": 8668,
          "num_classes": 14,
          "mean_labels": 0.6222888786340564,
          "macro_ap": 0.07165501291778696,
          "micro_ap": 0.07494816411558966,
          "macro_auroc": 0.6117913342740502,
          "ap_classes": 14,
          "auroc_classes": 14,
          "per_class": {
            "Atelectasis": {
              "positives": 828,
              "prevalence": 0.09552376557452699,
              "ap": 0.11406821675517238,
              "auroc": 0.5645894952183773
            },
            "Cardiomegaly": {
              "positives": 170,
              "prevalence": 0.01961236732810337,
              "ap": 0.063934936016878,
              "auroc": 0.6764429000595296
            },
            "Consolidation": {
              "positives": 285,
              "prevalence": 0.03287955699123212,
              "ap": 0.053432399586558744,
              "auroc": 0.6245701932273126
            },
            "Edema": {
              "positives": 137,
              "prevalence": 0.015805260729118597,
              "ap": 0.04323925949332959,
              "auroc": 0.7771579306727633
            },
            "Effusion": {
              "positives": 866,
              "prevalence": 0.09990770650669129,
              "ap": 0.15376623960987112,
              "auroc": 0.6094579290085504
            },
            "Emphysema": {
              "positives": 142,
              "prevalence": 0.016382095062298107,
              "ap": 0.013196131286465022,
              "auroc": 0.4241115824668867
            },
            "Fibrosis": {
              "positives": 125,
              "prevalence": 0.01442085832948777,
              "ap": 0.029595122583696495,
              "auroc": 0.6046278824768816
            },
            "Hernia": {
              "positives": 14,
              "prevalence": 0.0016151361329026304,
              "ap": 0.007453301245034342,
              "auroc": 0.598818878140579
            },
            "Infiltration": {
              "positives": 1378,
              "prevalence": 0.15897554222427318,
              "ap": 0.2411387692525531,
              "auroc": 0.6123310457692009
            },
            "Mass": {
              "positives": 403,
              "prevalence": 0.046492847254268575,
              "ap": 0.09076218521046911,
              "auroc": 0.6258993723720613
            },
            "Nodule": {
              "positives": 471,
              "prevalence": 0.05433779418550992,
              "ap": 0.08696189449888747,
              "auroc": 0.6024967189332124
            },
            "Pleural_Thickening": {
              "positives": 224,
              "prevalence": 0.025842178126442086,
              "ap": 0.04136165704149765,
              "auroc": 0.6244784441192394
            },
            "Pneumonia": {
              "positives": 88,
              "prevalence": 0.01015228426395939,
              "ap": 0.016839624190346444,
              "auroc": 0.611808645899555
            },
            "Pneumothorax": {
              "positives": 263,
              "prevalence": 0.03034148592524227,
              "ap": 0.047420444078258156,
              "auroc": 0.6082876614725528
            }
          }
        }
      }
    },
    "plant-pecore": {
      "protocol": "Exploratory calibration-only prompt comparison; no held-out evaluation",
      "data": {
        "dataset": "plant-pathology-2021",
        "revision": "e9c0d933b2013362ffa61eac42cc9cfd3ab1eb46",
        "split": "train",
        "fold": null,
        "eligible_images": 16768,
        "selected_images": 2048,
        "sampling": "lowest SHA256(42:image_id), no label stratification",
        "prior_calibration_matching_ids": 118
      },
      "model": "PE-Core-L-14-336",
      "pretrained": "meta",
      "context_length": 32,
      "amp": "bfloat16 image; float32 text/scoring",
      "classnames": [
        "complex",
        "frog_eye_leaf_spot",
        "healthy",
        "powdery_mildew",
        "rust",
        "scab"
      ],
      "results": {
        "baseline": {
          "score": "cosine",
          "max_prompt_tokens": 17,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.4793523758721289,
          "micro_ap": 0.4395826479785424,
          "macro_auroc": 0.7970768020218041,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.23471235616531197,
              "auroc": 0.7308557773796384
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.3668434454723716,
              "auroc": 0.7208469780723586
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.6694328715603408,
              "auroc": 0.8784652798692365
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.6922756567254592,
              "auroc": 0.9093996123554023
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.4345230847566603,
              "auroc": 0.8257068573285669
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.47832684055262925,
              "auroc": 0.717186307125622
            }
          }
        },
        "baseline_paired": {
          "score": "paired",
          "max_prompt_tokens": 17,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.5545578286543825,
          "micro_ap": 0.5104833523593981,
          "macro_auroc": 0.8069023327844739,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.2209536118373001,
              "auroc": 0.7168172427823499
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.3816034093707175,
              "auroc": 0.756289460585261
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.9806509982719128,
              "auroc": 0.9942047982890357
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.6692966855199836,
              "auroc": 0.8743193145459021
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.7414652918009526,
              "auroc": 0.9145526718974883
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.33337697512542874,
              "auroc": 0.5852305086068064
            }
          }
        },
        "domain_templates": {
          "score": "cosine",
          "max_prompt_tokens": 17,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.4800183734421088,
          "micro_ap": 0.4390955511281545,
          "macro_auroc": 0.7978576467653657,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.2310305855038574,
              "auroc": 0.7278210261056047
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.3721137942967519,
              "auroc": 0.7262083780880774
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.6702427037144967,
              "auroc": 0.8804169148776679
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.6807130807744001,
              "auroc": 0.9050809131183855
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.44475511570324655,
              "auroc": 0.8295719392729468
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.4812549606599001,
              "auroc": 0.7180467091295117
            }
          }
        },
        "visual_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 22,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.5336487457599036,
          "micro_ap": 0.46244270593936876,
          "macro_auroc": 0.814837348053255,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.2520662842003437,
              "auroc": 0.7428284331466775
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.3703584791200044,
              "auroc": 0.7301472321919782
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.6897650070882897,
              "auroc": 0.8958893604713642
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.7458814453043233,
              "auroc": 0.9281896074329313
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.6479826888092872,
              "auroc": 0.8677700914619991
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.4958385700371735,
              "auroc": 0.72419936361458
            }
          }
        },
        "baseline_plus_descriptions": {
          "score": "cosine",
          "max_prompt_tokens": 22,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.5177837920957197,
          "micro_ap": 0.45739925848263646,
          "macro_auroc": 0.8098908827942314,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.24583659127685337,
              "auroc": 0.7389169759490342
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.36906332451817847,
              "auroc": 0.7281575017683581
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.688142392624935,
              "auroc": 0.8910573722315906
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.726334891502881,
              "auroc": 0.921598111001723
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.5865548455557366,
              "auroc": 0.8561075258832814
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.49077070709573356,
              "auroc": 0.7235078099314018
            }
          }
        },
        "descriptions_paired": {
          "score": "paired",
          "max_prompt_tokens": 22,
          "num_images": 2048,
          "num_classes": 6,
          "mean_labels": 1.08740234375,
          "macro_ap": 0.5824159522237234,
          "micro_ap": 0.5282309449609012,
          "macro_auroc": 0.8202526482711723,
          "ap_classes": 6,
          "auroc_classes": 6,
          "per_class": {
            "complex": {
              "positives": 247,
              "prevalence": 0.12060546875,
              "ap": 0.21851572029358318,
              "auroc": 0.7194608483366192
            },
            "frog_eye_leaf_spot": {
              "positives": 490,
              "prevalence": 0.2392578125,
              "ap": 0.3803215487004136,
              "auroc": 0.7479526341987373
            },
            "healthy": {
              "positives": 481,
              "prevalence": 0.23486328125,
              "ap": 0.9792282449224147,
              "auroc": 0.9938187168563684
            },
            "powdery_mildew": {
              "positives": 136,
              "prevalence": 0.06640625,
              "ap": 0.7558624530620699,
              "auroc": 0.9058923517105587
            },
            "rust": {
              "positives": 238,
              "prevalence": 0.1162109375,
              "ap": 0.7989922587335336,
              "auroc": 0.9304076326663262
            },
            "scab": {
              "positives": 635,
              "prevalence": 0.31005859375,
              "ap": 0.36157548763032554,
              "auroc": 0.6239837058584238
            }
          }
        }
      }
    }
  },
  "verification": {
    "bigearth-pecore": {
      "matching_prior_baseline_images": 8192,
      "max_score_difference": 0.0008774697780609131,
      "mean_score_difference": 3.571769866539398e-06,
      "baseline_macro_ap_difference": 7.119980874659149e-06,
      "min_class_positives": 25,
      "unique_patients": 0
    },
    "nih-biomedclip": {
      "matching_prior_baseline_images": 8668,
      "max_score_difference": 1.7881393432617188e-07,
      "mean_score_difference": 2.9685178759564224e-08,
      "baseline_macro_ap_difference": -7.476390090355167e-08,
      "min_class_positives": 14,
      "unique_patients": 2797
    },
    "nih-pecore": {
      "matching_prior_baseline_images": 8668,
      "max_score_difference": 0.0007936060428619385,
      "mean_score_difference": 3.085690650550532e-06,
      "baseline_macro_ap_difference": -2.882140166695496e-05,
      "min_class_positives": 14,
      "unique_patients": 2797
    },
    "nih-siglip2": {
      "matching_prior_baseline_images": 8668,
      "max_score_difference": 0.0014963746070861816,
      "mean_score_difference": 1.1535715202626307e-05,
      "baseline_macro_ap_difference": -2.832783092478891e-05,
      "min_class_positives": 14,
      "unique_patients": 2797
    },
    "plant-pecore": {
      "matching_prior_baseline_images": 118,
      "max_score_difference": 0.0,
      "mean_score_difference": 0.0,
      "baseline_macro_ap_difference": 0.0,
      "min_class_positives": 136,
      "unique_patients": 0
    }
  }
}