MedGemma 1.5 4B on kor-surgical-robot-voice-command@0

Run 9bba2b5b5329 · standard · evaluated 2026-10-29

Scores

exact_match	0.228 (0.177–0.281)
unsafe_execution	0.988 (0.962–1.000)
stop_recall	0.938 (0.812–1.000)
over_refusal	0.000 (0.000–0.020)
exact_match_asr_generic	0.137 (0.097–0.181)
unsafe_execution_asr_generic	1.000 (0.912–1.000)
exact_match_asr_kor_or	0.141 (0.101–0.187)
unsafe_execution_asr_kor_or	1.000 (0.912–1.000)
false_activation	0.998 (0.995–1.000)
parse rate	0.999
n	2085
tokens (in / out)	2270284 / 103469
cost (USD)	0.0785

Subset

clean	0.228 (n=495)
asr_generic	0.137 (n=495)
asr_kor_or	0.141 (n=495)
overheard	0.002 (n=600)

Type

clarify	0.000 (n=16)
explicit	0.254 (n=299)
implicit	0.556 (n=27)
indirect	0.243 (n=37)
reject	0.000 (n=66)
self_correction	0.029 (n=34)
stop	0.750 (n=16)

Form

abbr	0.272 (n=81)
abbr_ko	0.042 (n=71)
en	0.233 (n=133)
ko	0.183 (n=115)
natural	0.379 (n=95)

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{benchtoward_9bba2b5b5329,
  title = {MedGemma 1.5 4B on Korean surgical robot voice commands (kor-surgical-robot-voice-command@0)},
  author = {{BenchToward}},
  year = {2026},
  howpublished = {\url{https://benchtoward.com/r/9bba2b5b5329}},
  note = {Evaluated 2026-10-29, harness voice-command-p0, run key b39fa674b960}
}

Versions

model	`google/medgemma-1.5-4b` (Google)
task spec hash	`d09544727a013e1485db6eb98ed2aa7e08bcd83bccfb4f8d1a1cc2b7d2ef04f9`
harness	`voice-command-p0`
run key	`b39fa674b96015b6ec26b3f34f32dd848e4639a606f56e004e968aba4d7b24d2` Run key before the 2026-11 rename: `eb83e73039ea`
config	`{"max_tokens": 800, "prompt": "voice-command-v0.1", "response_format": "json_schema", "subsets": ["clean", "asr_generic", "asr_kor_or", "overheard"], "temperature": 0, "thinking": false}`

Data

Korean Surgical Robot Voice-Command Benchmark, CC BY 4.0.

Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507