About & methods
What this is
CheXboard reports how published chest-radiograph classifiers perform when each is run, unmodified and without site-specific fine-tuning, over the same multinational collection of cohorts. Every model is evaluated on the findings it was released to predict; a model is scored on a cohort only where that cohort carries a label for the finding, so every cohort × finding score compares like with like. Models cover different numbers of findings, so each average is shown with the count it is taken over, and the Findings filter or the Compare tab gives a matched comparison. Numbers here are the same ones reported in the accompanying manuscript.
Models
VLM = vision–language model · CNN = supervised convolutional classifier · GEN = generative report model · GEN+ = generative model with a trained probability booster.
Cohorts
Sample counts are the largest number of evaluated frontal radiographs carrying a usable label for any single finding in that cohort; a finding labelled in only part of the cohort is scored on fewer. Private cohorts contributed evaluation results only — no images or patient-level data are distributed by this site or its repository.
Metrics
How the numbers are pooled
Each raw cell is one model × cohort × finding × metric. The default per-finding aggregation averages across cohorts within a finding first, then across findings — so a finding labelled in many cohorts does not outweigh a rare one, and this is the aggregation the manuscript's headline figures use. Per dataset reverses the order (average findings within a cohort, then cohorts), and all cells takes a plain mean, which is biased toward the largest cohorts.
Confidence intervals on an aggregate are the spread across the averaged units (mean ± 1.96·SE) when at least two units contribute. When a filter narrows the view to a single unit that spread collapses to zero, so the interval falls back to the contributing cells' own bootstrap CIs from the summary — a filtered view still shows real uncertainty rather than a point.
Caveats
- Binary output. CheXagent emits binary labels only, so it appears under the Youden sensitivity and specificity metrics and is absent from the probability-based metrics (AUROC, AUPRC, operating-point).
- AUROC axis floor. Charts start the AUROC axis at 0.5 (chance); a below-chance run therefore sits on the floor rather than extending below it.
- Age bands. Only cohorts carrying per-image ages contribute to the <18 / 18–65 / 65+ views, so those views cover fewer cohorts and findings than Overall. Two cohorts are adult-only and have no <18 band.
- One inference pass. Intervals describe test-set sampling variability, not run-to-run or seed variability; each model was run once per cohort.
- Labels are the reference, not truth. Cohort labels come from each site's own annotation or report-extraction process and differ in definition and rigour; some disagreement across cohorts reflects labelling rather than model behaviour.
Reproducibility
The site is a pure derivative of a single summary table produced by the analysis pipeline; it never recomputes anything in the browser beyond the pooling described above. The build stamps the source signature below into the data files, so any figure here can be traced to an exact upstream state.
How to cite
Please cite the arXiv preprint. This entry is updated once the article is published.