# BenchToward: guide for AI agents

Version: p0 (2026-10-29). Base URL: `https://benchtoward.com`

BenchToward evaluates medical AI models on open datasets from Seoul National University Hospital (SNUH), once per model and task, and publishes scores, per-item predictions and exact versions so that nobody has to repeat the same experiment. Everything on the website is also available as JSON under `/v1`. Machine-readable spec: `https://benchtoward.com/v1/openapi.json`.

## Rule one: look up before you run

Before you evaluate a model on a BenchToward task yourself, ask whether the result already exists:

```bash
curl "https://benchtoward.com/v1/lookup?model=google/medgemma-27b&task=clinicalqa@0"
# {"status": "evaluated", "result": {...}}   or   {"status": "not_evaluated", "result": null}
```

`model` is the model id shown in results, `<developer>/<model>` (`google/medgemma-27b`, `anthropic/claude-haiku-5.5`). `task` accepts `slug@version` or `slug` (latest version). If the result exists, cite it and download its predictions instead of spending compute.

## Endpoints

All responses are JSON. No token is needed in p0.

| Route | Returns |
|---|---|
| `GET /v1/datasets` | Datasets with license, license URL and citation |
| `GET /v1/tasks` | Tasks: slug, version, dataset, kind, primary metric, preview flag, number of results |
| `GET /v1/tasks/{slug}` | Full task specification (`?version=` for a specific version) |
| `GET /v1/leaderboard?task=<slug[@version]>&model_type=&limit=` | Ranked rows for one task |
| `GET /v1/results?dataset=&task=&model=&model_type=` | Results matching all given filters |
| `GET /v1/results/{run_id}` | One result: all metrics with 95% CIs, configuration, versions, cost, BibTeX, predictions URL |
| `GET /v1/lookup?model=&task=` | Whether a model has been evaluated on a task |
| `GET /v1/stats` | Counts, total cost, number of reuses (lookups that found a result, plus downloads) |
| `GET /v1/papers?status=validated\|queued` | Papers the BenchToward model evaluator validated or has queued, with task, metric and rank |
| `GET /v1/papers/{doi}` | One paper: our summary, model at a glance, results on SNUH data (DOIs contain slashes; use them as they are) |
| `GET /v1/agent/now` | What the model evaluator is doing right now: paper, stage (1-7), recent log lines, queue; `building` counts active benchmark builds |
| `GET /v1/builds` | Benchmarks the BenchToward benchmark builder is building, reviewing or has queued: stage (0-6), progress, adopted protocol, KHDP dataset status |
| `GET /artifacts/{run_id}/predictions.parquet` | Per-item predictions for one result |

Examples:

```bash
# Leaderboard for one task, LLMs only
curl "https://benchtoward.com/v1/leaderboard?task=clinicalqa@0&model_type=llm"

# One result with metrics, versions and BibTeX
curl "https://benchtoward.com/v1/results/<run_id>"

# Per-item predictions (columns: instance_id, prediction or score, label, correct)
curl -o preds.parquet "https://benchtoward.com/artifacts/<run_id>/predictions.parquet"
```

A leaderboard row looks like:

```json
{"rank": 1, "run_id": "0123456789ab", "task": "clinicalqa@0",
 "model": {"id": "google/medgemma-27b", "name": "MedGemma 27B", "type": "llm", "developer": "Google"},
 "tier": "standard", "metric": {"name": "accuracy", "value": 0.71, "ci_low": 0.68, "ci_high": 0.74},
 "n": 1045, "parse_rate": 0.99, "cost_usd": 0.21, "eval_date": "2026-10-29",
 "url": "https://benchtoward.com/r/0123456789ab"}
```

## Tasks in p0

| Task | Data | What is scored |
|---|---|---|
| `clinicalqa@0` | ClinicalQA, 1,045 Korean KMLE-level five-option questions | Accuracy; unparsable answers count as wrong |
| `vitaldb-ioh@0` | VitalDB, numeric mean arterial pressure every 2 s | AUROC for hypotension (MAP < 65 mmHg for ≥ 60 s) in the 5 minutes after a 5-minute window; patient-level split |
| `inspire-mortality@0` | INSPIRE 1.3, one operation per patient: age, sex, emergency, BMI, anesthesia duration, ASA class, 13 preoperative labs | AUROC for in-hospital death within 30 days after leaving the operating room; cohort, inputs and train/test split from the INSPIRE authors' `gbm_mortality.py` (deviations in the spec) |
| `vitaldb-arrhythmia-abnormal@0` | VitalDB Arrhythmia Database, 5-second ECG lead II segments at 500 Hz | AUROC for abnormal rhythm (any rhythm label other than normal sinus rhythm), as in the dataset paper's technical validation; Noise not scored; patient-level split shared with other VitalDB tasks |
| `k-or-speech-asr@0` | Synthetic K-OR Speech Audio, 8,000 clips (2,000 Korean operating-room utterances × 4 synthetic voices) | Character error rate (lower is better) after normalisation `kor-norm-0` (in the task spec); WER too; intervals from a bootstrap over utterances; results by voice, code-switching style, intent and speaker role |
| `kor-surgical-robot-voice-command@0` | Korean Surgical Robot Voice-Command Benchmark 1.0.0: 2,085 items in 4 subsets (clean text, two ASR versions of it, overheard operating-room talk), each with a scene prompt | Exact match of the robot command on the 495 clean items (response type, action sequence and every slot); also unsafe execution, stop recall and over-refusal (clean), false activation on overheard talk, and exact match and unsafe execution per ASR subset. Unsafe execution, over-refusal and false activation are lower is better. `n` counts all 2,085 items; intervals from a bootstrap over wording variants (Clopper-Pearson at 0% or 100%) |

All are **preview** versions and may change. A result always names the exact task version and specification hash it was computed on.

## Datasets, licenses and citation

- **ClinicalQA**: CC BY 4.0 on KHDP (upstream Apache-2.0). Cite: SNUH-HARI. ClinicalQA: Korean Medical Knowledge QA Benchmark. Seoul National University Hospital Healthcare AI Research Institute, 2025. doi:10.23421/2026.0800. It has been public on Hugging Face since 2025-04-04; models trained after that may have seen it.
- **Synthetic K-OR Speech Audio**: Apache-2.0 with a use restriction. The audio may be used for speech recognition research, benchmarking, evaluation and zero-shot testing; it must not be used to train, fine-tune or develop text-to-speech, voice-cloning or speaker-identification systems, nor to create any derivative voice asset, and redistribution must keep this restriction. Cite: Seoul National University Hospital, Department of Anesthesiology and Pain Medicine. Synthetic K-OR Speech Audio (version 1.0.0). KHDP. 2026. doi:10.23421/2026.0774. The speech is synthetic, so error rates understate those in real operating rooms.
- **Korean Surgical Robot Voice-Command Benchmark**: CC BY 4.0 on KHDP (`kor-surgical-robot-voice-command` 1.0.0). Cite: Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507. The 600 overheard items come from the Synthetic K-OR Speech Corpus v1.0 (doi:10.23421/2026.0773, CC BY 4.0); 113 of them are derived from MM-OR (Özsoy et al., CVPR 2025; Apache-2.0) and keep that attribution. Gold labels were written with LLM assistance and have not been reviewed by surgeons.
- **VitalDB**: CC BY 4.0. Cite: Lee HC, Park Y, Yoon SB, Yang SM, Park D, Jung CW. VitalDB, a high-fidelity multi-parameter vital signs database in surgical patients. Sci Data. 2022;9:279. doi:10.1038/s41597-022-01411-5.
- **INSPIRE**: CC BY 4.0 on KHDP (`inspire` 1.3, doi:10.13026/46m4-f655). Cite: Lim L, Lee H, Jung CW, Sim D, Borrat X, Pollard TJ, Celi LA, Mark RG, Vistisen ST, Lee HC. INSPIRE, a publicly available research dataset for perioperative medicine. Sci Data. 2024;11:655. doi:10.1038/s41597-024-03517-4. Measurements are binned to 5th-percentile values, so inputs are ordinal categories.
- **VitalDB Arrhythmia Database**: CC BY 4.0. Cite: Eun D, Shim K, Lee H, Lim Y, Lim H, Lee H, Lee HC. VitalDB Arrhythmia Database: An Anesthesiologist-Validated Large-Scale Intraoperative Arrhythmia Dataset with Beat and Rhythm Labels (version 1.0.0). PhysioNet. 2026. doi:10.13026/axd6-wm13. The ECG waveforms come from VitalDB (cite VitalDB as well).
- **SCOPE** (KHDP `icu-cardiac-arrest` 1.0): CC BY 4.0. Cite: SCOPE: A survival and cardiac arrest outcome dataset with PPG and ECG waveforms from intensive care units (version 1.0). KHDP; 2025. doi:10.23421/2025.0698.

BenchToward results and per-item predictions are released under CC BY 4.0. When you use a BenchToward result, cite the dataset and the result (BibTeX is in `/v1/results/{run_id}`).

## Papers and the live agent

An agent reads new papers with released code and weights (npj Digital Medicine first, arXiv next), picks the open SNUH dataset that fits, runs the authors' model and publishes the result as a preview with tier `reference`. When no existing task fits, it defines one from the paper's methods; such tasks are `version 0`, marked preview, and list any `deviations` from the paper in their spec. Papers it could not run are not listed. Watch it at `https://benchtoward.com/live`.

## Tags (field `tier`)

Each result says who ran the model:

- **official**: run from what the model's authors submitted (not open yet).
- **standard**: run by BenchToward with the standard prompt and harness, identical for every model on the task.
- **agent**: run by the BenchToward model evaluator with the authors' released weights, on open SNUH data. Not a verdict on the paper: the agent's setup can differ from the authors'.
- **baseline**: a simple comparison model built by BenchToward (majority class, logistic regression).

## Not yet available

Submitting models, API tokens and the `benchtoward` CLI are planned. Until then, to request an evaluation, open an issue or contact the maintainers listed on https://benchtoward.com/about.

## Errors

Errors are JSON with a machine-readable code: `{"error": "task_not_found", "message": "..."}`. Codes: `task_not_found`, `run_not_found`.

BenchToward is an independent project using open SNUH datasets. It is not an official service of Seoul National University Hospital.