On 17 November 2026, Kiro described an evaluation failure in which a new model achieved an expected pass rate on Terminal Bench 2.1 while some tools failed on 40% of their calls. Kiro says the model submitted invalid inputs because a tool description that worked for other models did not work well for the new one. The post does not identify the model or a fixed software version.[1]

According to Kiro, the model could retry until it submitted a valid request, so the errors did not affect the pass rate. Those retries consumed tokens and increased latency. The company says the incident showed how a benchmark nearing saturation, where tasks are solved despite regressions, can miss changes that matter to users.[1]

Kiro says it now maintains a versioned benchmark suite informed by aggregated usage categories from internal Amazon users. Its process tracks pass or fail, signals such as latency and token use, and large language model judgments of harder-to-measure qualities such as code quality and response tone. It also compares benchmark signals with surveys, satisfaction ratings and user feedback. Kiro reports that tool-failure data helped it fix the inconsistent description, but the post does not provide results that independently establish the fix's effect.[1]

The post also describes a curation problem: a highly rated, merged pull request looked suitable as a benchmark example until repository history showed that the next commit fixed bugs it had introduced. Kiro says it plans further research on benchmark curation and complex customer configurations.[1]

Sources

  1. Lessons learned from one million evaluationskiro.dev · 17 Nov 2026

Check or correct this page

Verification record

The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.

c2f831f7735b42384f6fa267dc6a7597fd4fe1ad74f51d47e16df88658c00ec9