On 16 November 2026, Amazon announced BenchProofer, an open-source tool that adds formal-verification artifacts to coding benchmarks. The company also released SWE-Proof, which applies the approach to all 500 tasks in SWE-Bench Verified. The aim is to expose errors that a benchmark’s existing tests might miss.[1]
For each task, BenchProofer uses a problem description, an accepted reference solution and a way to test fixes. It generates a formal specification, a reference implementation and a proof that the implementation satisfies the specification under stated assumptions. A verifier checks the proof; tests and audits assess how the implementation corresponds to the repository code. Contracts for existing functions called by new code are tested rather than formally verified, Amazon says.[1]
In audit mode, additional tests look for behavioral differences between a test-passing patch and the reference. In verified mode, an agent receives a specification and produces both a repository patch and a separate implementation for verification; an audit checks their correspondence. Amazon reports that its audit flagged 134 of 425 Claude Opus 4.8 patches that passed the original tests. Human reviewers found defects in 21 of a sample of 41 flagged patches; the other 20 were valid fixes.[1]
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
d31ae4f33a1031567a524f627f688bd1b9c5bf403c2293fd8bf79dccc0a8725d