Prior (ignores the input) on kor-surgical-robot-voice-command@0

Run a923656cf11c · baseline · evaluated 2026-10-29

Scores

exact_match	0.133 (0.088–0.192)
unsafe_execution	0.000 (0.000–0.088)
stop_recall	0.000 (0.000–0.206)
over_refusal	1.000 (0.980–1.000)
exact_match_asr_generic	0.133 (0.088–0.192)
unsafe_execution_asr_generic	0.000 (0.000–0.088)
exact_match_asr_kor_or	0.133 (0.088–0.192)
unsafe_execution_asr_kor_or	0.000 (0.000–0.088)
false_activation	0.000 (0.000–0.006)
parse rate	1.000
n	2085
tokens (in / out)	0 / 0
cost (USD)	0.0000

Subset

clean	0.133 (n=495)
asr_generic	0.133 (n=495)
asr_kor_or	0.133 (n=495)
overheard	1.000 (n=600)

Type

clarify	0.000 (n=16)
explicit	0.000 (n=299)
implicit	0.000 (n=27)
indirect	0.000 (n=37)
reject	1.000 (n=66)
self_correction	0.000 (n=34)
stop	0.000 (n=16)

Form

abbr	0.148 (n=81)
abbr_ko	0.141 (n=71)
en	0.143 (n=133)
ko	0.157 (n=115)
natural	0.074 (n=95)

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{benchtoward_a923656cf11c,
  title = {Prior (ignores the input) on Korean surgical robot voice commands (kor-surgical-robot-voice-command@0)},
  author = {{BenchToward}},
  year = {2026},
  howpublished = {\url{https://benchtoward.com/r/a923656cf11c}},
  note = {Evaluated 2026-10-29, harness p0.1, run key c268cc1d9af3}
}

Versions

model	`benchtoward/prior` (BenchToward)
task spec hash	`d09544727a013e1485db6eb98ed2aa7e08bcd83bccfb4f8d1a1cc2b7d2ef04f9`
harness	`p0.1`
run key	`c268cc1d9af3a15efa0309e7eb33de6b2087e40231b7321275fc542576b57e28` Run key before the 2026-11 rename: `0075c68ad562`
config	`{"output": {"actions": [], "message": null, "response": "reject"}, "rule": "most frequent (response, actions) of train/train_v0.2.jsonl", "subsets": ["clean", "asr_generic", "asr_kor_or", "overheard"]}`

Data

Korean Surgical Robot Voice-Command Benchmark, CC BY 4.0.

Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507