DeepSeek V4.1 Flash 552B-A16B on kor-surgical-robot-voice-command@0

Run da9bb98f3974 · standard · evaluated 2026-10-29

Scores

exact_match	0.929 (0.893–0.962)
unsafe_execution	0.159 (0.028–0.313)
stop_recall	1.000 (0.794–1.000)
over_refusal	0.019 (0.003–0.043)
exact_match_asr_generic	0.632 (0.583–0.685)
unsafe_execution_asr_generic	0.122 (0.029–0.245)
exact_match_asr_kor_or	0.646 (0.598–0.694)
unsafe_execution_asr_kor_or	0.110 (0.023–0.221)
false_activation	0.022 (0.010–0.033)
parse rate	1.000
n	2085
tokens (in / out)	2554034 / 90343
cost (USD)	0.0000

Subset

clean	0.929 (n=495)
asr_generic	0.632 (n=495)
asr_kor_or	0.646 (n=495)
overheard	0.975 (n=600)

Type

clarify	1.000 (n=16)
explicit	0.953 (n=299)
implicit	0.963 (n=27)
indirect	0.838 (n=37)
reject	0.803 (n=66)
self_correction	0.971 (n=34)
stop	1.000 (n=16)

Form

abbr	0.889 (n=81)
abbr_ko	0.930 (n=71)
en	0.940 (n=133)
ko	0.913 (n=115)
natural	0.968 (n=95)

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{benchtoward_da9bb98f3974,
  title = {DeepSeek V4.1 Flash 552B-A16B on Korean surgical robot voice commands (kor-surgical-robot-voice-command@0)},
  author = {{BenchToward}},
  year = {2026},
  howpublished = {\url{https://benchtoward.com/r/da9bb98f3974}},
  note = {Evaluated 2026-10-29, harness voice-command-p0, run key c12126153ddf}
}

Versions

model	`deepseek/deepseek-v4.1-flash-552b-a16b` (DeepSeek)
task spec hash	`d09544727a013e1485db6eb98ed2aa7e08bcd83bccfb4f8d1a1cc2b7d2ef04f9`
harness	`voice-command-p0`
run key	`c12126153ddfc3803ad95d4e3d5dc4c6d822dad0f4861aab1eb420b6b7fe183f` Run key before the 2026-11 rename: `5ae276e7ec72`
config	`{"max_tokens": 800, "prompt": "voice-command-v0.1", "response_format": "json_schema", "subsets": ["clean", "asr_generic", "asr_kor_or", "overheard"], "temperature": 0, "thinking": false}`

Data

Korean Surgical Robot Voice-Command Benchmark, CC BY 4.0.

Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507