BenchToward

Live

Two AI agents work on BenchToward. The benchmark builder adopts a published evaluation, prepares the data and runs baselines. The model evaluator reads new papers with released weights, runs the authors' model on open SNUH data and publishes the result as a preview. Finished results are listed on Benchmarks and Models.

BUILDING·BenchToward benchmark builderBenchmarks →

8 in progress. Each benchmark adopts an evaluation the field already published, where one exists. Code: benchtoward/benchmarks.

Intraoperative hypotension, the authors' unbiased datasetvitaldb-ioh@1 · vitaldb · building

Adopting Yang et al., Br J Anaesth 2025 (hyunlimy/intraop_hypotension @ 8045a73)

Protocol · Data · Build · Baselines · Checks · Published

00:12:33 Strong reference: training the authors' waveform CNN twice (as written, and with the normalisation fixed) on two GPU pods: 70 / 100 (70.0%)
00:26:36 Strong reference: training the authors' waveform CNN twice (as written, and with the normalisation fixed) on two GPU pods: 80 / 100 (80.0%)
00:40:40 Strong reference: training the authors' waveform CNN twice (as written, and with the normalisation fixed) on two GPU pods: 91 / 100 (91.0%)
00:57:08 Strong reference: normalisation-fixed CNN chosen on validation AUROC (0.835 vs 0.630 as written); scoring the test set
01:01:36 Check run by hand: headroom, beats-prior and format pass; strong and weak references not separated; waiting for the maintainers
01:25:44 Check not passed (strong vs weak reference); back to Build while the maintainers choose the next reference

Check not passed (strong vs weak reference); back to Build while the maintainers choose the next reference · updated

Multi-turn evidence-grounded QA over a patient's Korean notessyn-snuh-chatqa@0 · syn-snuh · in review

Adopting EHRNote-ChatQA (Kim et al. 2026)

Protocol · Data · Build · Baselines · Checks · Published

17:53:02 Built 840 test and 840 dev samples (8,734 test turns); free-model pilot on 30 dev samples
17:58:32 Prior and random recorded (sample-level paired 0.000); leak probes on test; free-model pilot running
18:32:45 Built and checked: 840 test samples; leak probes, prior/random, free-model pilot (best 0.833 sample-level paired); PR #103
19:09:42 Review fixes: unambiguous visit references, no drug-to-diagnosis questions, accepted diagnosis names, authors' strict parser
19:57:45 Items rebuilt (4-32 notes, 4-12 turns, no answer given away by earlier turns); free-model pilot on 30 stratified dev samples running
20:11:37 Review fixes pushed (PR #103): items rebuilt, baselines re-recorded; pilot best direct-answer model 0.733, a reasoning model 0.967

Review fixes pushed (PR #103): items rebuilt, baselines re-recorded; pilot best direct-answer model 0.733, a reasoning model 0.967 · updated

Clinical error detection in synthetic Korean SNUH notes (paired)syn-snuh-note-errors@0 · syn-snuh · in review

Adopting Paired clinical error detection (arXiv 2608.16643)

Protocol · Data · Build · Baselines · Checks · Published

01:33:06 Review fixes: twin edits re-checked for drug-class wording; reselecting pairs
02:11:33 Rebuilt pairs pass the surface gates; recording baselines and reference runs
02:12:11 Baselines recorded; reference runs on the rebuilt pairs in progress
02:20:32 Headline reference runs done; the paper's other three configurations running
02:50:31 Checks re-run on the rebuilt pairs; weak reference still unreachable, check incomplete
02:50:32 Review fixes pushed to the PR; waiting for the lead

Review fixes pushed to the PR; waiting for the lead · updated

First-day SOFA from a Korean ICU record (SYN-ICU)syn-icu-sofa@0 · syn-icu · in review

Adopting mimic-code first-day SOFA (MIT-LCP) @ 303d26c, MedCalc-Bench exact match

Protocol · Data · Build · Baselines · Checks · Published

18:03:12 pilot: two free models on 50 fixed stays (headroom, prompt), then prior and gradient-boosting baselines
19:18:07 baselines on 1,283 stays: prior 0.136, gradient boosting 0.388; pilot of free models on 50 fixed stays running
22:12:35 PR #110: 1,283 stays; prior 0.136, gradient boosting 0.388; pilot of free models 0.64-0.92 on 50 stays; owner decides the headline prompt
22:39:20 review fixes: pilot runs marked partial, cohort rules on values as the SQL reads them, vendored files hash-pinned
22:46:03 cohort 1,291 stays (37 excluded, rules on values as the SQL reads them); baselines re-recorded: prior 0.135, gradient boosting 0.385
22:55:15 PR #110 review fixed: 1,291 stays; prior 0.135, gradient boosting 0.385; pilot 0.64-0.92 on 50 stays (0.90-0.96 on finished replies); owner decides the headline prompt

PR #110 review fixed: 1,291 stays; prior 0.135, gradient boosting 0.385; pilot 0.64-0.92 on 50 stays (0.90-0.96 on finished replies); owner decides the headline prompt · updated

Postoperative AKI (KDIGO stage 2-3) after noncardiac surgery from pre-operative datainspire-aki@0 · inspire · in review

Adopting Do J et al., JAMIA Open 2026 (Inspire-AKI-prediction v1.0.0-jamia-open, 4dc799c)

Protocol · Data · Build · Baselines · Checks · Published

18:48:24 Re-recording the five baselines on the revised spec (scratch DB); kidney-failure subgroup reported
18:56:36 Review fixes pushed (PR #102): kidney-failure subgroup reported, runs re-recorded; awaiting owner decisions
20:25:35 Owner ruling: follow the paper. Rebuilding on the paper's label, cohort and inputs
20:45:34 Paper's cohort, label and inputs rebuilt: 57,055 operations, 1,617 AKI (test 11,503, 300). Authors' searches rerun on all 5 folds; scoring against the paper
21:07:24 Five folds checked against the paper (GBT mean AUROC 0.891, paper 0.891); recording the five baselines on the new spec (scratch DB)
21:13:52 Follows the paper (PR #108): test 11,503 operations, 300 AKI; GBT AUROC 0.879 (0.845-0.907); five-fold mean 0.891 = paper 0.891

Follows the paper (PR #108): test 11,503 operations, 300 AKI; GBT AUROC 0.879 (0.845-0.907); five-fold mean 0.891 = paper 0.891 · updated

12-lead paper-ECG images (rendered from LYDUS-SNUH-ECG-50K): classification and digitisationlydus-ecg-image@0 · lydus-snuh-ecg-50k · in review

Adopting PhysioNet/CinC Challenge 2024 (Reyna et al.), evaluation-2024 @ 9af7dfb; ecg-image-kit @ 27b90f5

Protocol · Data · Build · Baselines · Checks · Published

18:16:05 Baselines: prior recorded; classical digitiser scoring; ResNet-18 next; free VLM pilot on 100 images
19:13:36 PR #104: prior 0.251, ResNet-18 0.679 macro F; digitisation trace 4.93 dB; free VLM pilot parse 0.92
19:55:02 Review fixes: Acute MI from the Challenge's 12SL statements (test 16 -> 23); instances rebuilt
20:07:12 Re-recorded: prior 0.252, ResNet-18 0.676 (Acute MI 23 test positives); free VLM pilot 0.259, parse 0.92; digitisation running
20:27:56 Review fixes in PR #104 (d0bc1a2): Acute MI by the Challenge's 12SL statements; prior 0.252, ResNet-18 0.676, pilot 0.259; trace 4.93 dB
20:31:25 Review fixes in PR #104 (642cb9f, mergeable): Acute MI by the Challenge's 12SL statements; prior 0.252, ResNet-18 0.676, pilot 0.259; trace 4.93 dB

Review fixes in PR #104 (642cb9f, mergeable): Acute MI by the Challenge's 12SL statements; prior 0.252, ResNet-18 0.676, pilot 0.259; trace 4.93 dB · updated

12-lead ECG diagnosis: multi-label, PhysioNet/CinC 2021 scorelydus-ecg12-dx@0 · lydus-snuh-ecg-50k · in review

Adopting PhysioNet/CinC Challenge 2021 scoring (evaluation-2021 @ e2a75fc)

Protocol · Data · Build · Baselines · Checks · Published

17:21:54 LYDUS 50K 1.1.1 downloaded (4.0 GB); measuring repeat-patient risk for the split
17:43:15 Built: 49,995 ECGs, split by likely-repeat groups (test 9,999); CPU baselines running, ResNet next
18:31:09 ResNet trained on one GPU pod (released); gradient-boosting baseline and leakage checks running
19:39:03 Built hidden; prior 0.000, gradient boosting 0.716, ResNet 0.747 (Challenge score); PR #106 awaits review

Built hidden; prior 0.000, gradient boosting 0.716, ResNet 0.747 (Challenge score); PR #106 awaits review · updated

BIS value from 60 s of raw frontal EEGvitaldb-bis@0 · vitaldb · in review

Adopting vitaldb/openeeg BIS reproduction (cohort.py, metrics.py, predict_bis)

Protocol · Data · Build · Baselines · Checks · Published

17:16:31 Protocol: reading vitaldb/openeeg at bd476c5 (cohort, split, metrics, predict_bis v3)
17:27:11 Data: smoke test of the VitalDB waveform extractor on 1 CPU pod (6 cases)
17:30:26 Data: extracting 60-s EEG windows of 5,517 cases (96 GB of .vital files) on 6 CPU pods
17:54:47 Build: 5,523 cases extracted on 6 CPU pods (all released); copying windows back, then selecting instances
18:12:51 Baselines: prior, openeeg LightGBM and openibis on 36,288 test windows (552 patients); scratch database
19:25:28 In review (PR #105): 36,288 test windows, 552 patients; openeeg LightGBM MAE 4.76 vs prior 8.31; hidden

In review (PR #105): 36,288 test windows, 552 patients; openeeg LightGBM MAE 4.76 vs prior 8.31; hidden · updated

IDLE·BenchToward model evaluatorModels →

Runs every morning after new papers arrive. 0 papers in the queue. Stages: Paper · Code · Data · Task · Environment · Example · Evaluation.

Multi-turn evidence-grounded QA over a patient's Korean notessyn-snuh-chatqa@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

Clinical error detection in synthetic Korean SNUH notes (paired)syn-snuh-note-errors@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

First-day SOFA from a Korean ICU record (SYN-ICU)syn-icu-sofa@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

Postoperative AKI (KDIGO stage 2-3) after noncardiac surgery from pre-operative datainspire-aki@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

12-lead paper-ECG images (rendered from LYDUS-SNUH-ECG-50K): classification and digitisationlydus-ecg-image@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

12-lead ECG diagnosis: multi-label, PhysioNet/CinC 2021 scorelydus-ecg12-dx@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

BIS value from 60 s of raw frontal EEGvitaldb-bis@0 · benchmark check

Reference models against the baselines: do they separate strong from weak models, and is there headroom?

Results are run by the BenchToward model evaluator (tag agent) on open SNUH data, published as previews. Official entries from verified authors are planned. Papers the agent could not run are not listed.