About
When a new medical AI model appears, many groups evaluate it on the same public datasets, each paying for the same compute and each with slightly different task definitions. BenchToward evaluates each model on each benchmark once, under a fixed and versioned definition, and publishes the result so that others can cite it, download the predictions and build on them instead of repeating the run. Over time, models submitted by their own authors and scored under identical conditions will form an official leaderboard.
How results are produced
- Each benchmark has a versioned specification; its hash is shown on the benchmark page. Benchmarks marked preview may still change.
- Each result is identified by a run key: a hash of the model version, the benchmark specification, the harness version and the run configuration. The same key is never computed twice.
- LLMs are queried with a fixed prompt at temperature 0. Answers that cannot be parsed count as wrong; the parse rate is reported.
- Confidence intervals are 95% bootstrap intervals over the unit each benchmark names (questions, patients, ICU stays or utterances), so items from one patient are resampled together.
- Ranks follow the score. Close scores with overlapping 95% intervals may not differ in a meaningful way: hover over a rank to see the 95% range of ranks a model reaches when all models are rescored on the same resampled items (a paired bootstrap). The API also returns this range.
How to read the tags
Each result carries a tag that says who ran the model.
| standard | Run by BenchToward with the standard prompt and harness, identical for every model on the benchmark (for example the LLMs on ClinicalQA). |
|---|---|
| agent | Run by the BenchToward model evaluator with the weights the authors released with their paper, on open SNUH data. The evaluator's setup can differ from the authors', so an agent result is not a verdict on the paper (see For authors). |
| baseline | A simple comparison model built by BenchToward, such as always choosing the most common answer or a logistic regression, so other scores have a floor to compare against. |
| official | Planned: run from code or a container the model's authors submit themselves, with the authors' identity verified. Submissions are not open yet. |
Benchmarks and models also show where they stand.
| building | The benchmark builder is preparing the protocol, data, code or baselines. |
|---|---|
| in review | Built and waiting for its check: the model evaluator compares reference models with the baselines, then the maintainers decide whether to publish. |
| published | Public, with a leaderboard. A preview benchmark is published but may still change as a new version. |
| evaluating | The model evaluator is reading the paper, setting up the released model or running it. |
For authors
Benchmarks here adopt evaluations that authors published, and the model evaluator runs released models without their authors' involvement, so our setup can differ from yours. We would rather get it right with you than on our own. You are welcome to:
- propose changes through a pull request or an issue on benchtoward/benchmarks: the task definition, preprocessing, splits, metrics, or how your model is loaded and called. Accepted fixes ship as a new benchmark version with a note of what changed, and the earlier version stays visible;
- provide your official code or container. We use it in place of our reimplementation and credit you on the benchmark page;
- ask us to withdraw a result while it is being checked. Open an issue and we take it down first and look into it afterwards.
All evaluation code is public under Apache-2.0, and code adopted from authors keeps their licence and credit.
Reuse
Every result page links per-item predictions and a BibTeX entry. Agents should call /v1/lookup before running an evaluation themselves.
The reuse count on the home page adds two things. Each /v1/lookup call that finds an existing result counts once. Downloads also count: a listed result's predictions file or result JSON (/v1/results/<run_id>), and a benchmark's leaderboard CSV. Lookups and downloads by bots are not counted, and repeat downloads of the same file by one visitor on the same day (Korea time) count once.
Code and data
Code is on GitHub: benchtoward/benchmarks. All datasets are open datasets from Seoul National University Hospital, distributed through KHDP; datasets that BenchToward builds are uploaded there with a DOI. Each benchmark page credits its dataset and links its licence.
Privacy
Visits are counted without cookies. An IP address is combined with a secret and the date and hashed; the address itself is never stored, and the hash changes every day. Downloads counted as reuses are stored with the same daily hash. Each page also sends one small request saying which page was opened, so that real browser visits can be told apart from automated fetches.
Disclaimer
BenchToward is an independent project using open SNUH datasets. It is not an official service of Seoul National University Hospital. We report performance on retrospective and synthetic clinical data; results are evidence, not clinical certification.