METR, a nonprofit that evaluates frontier AI models, said on 26 June 2026 that it considers none of its estimates for OpenAI's GPT-5.6 Sol robust. The reason, METR said, is that its pre-deployment measurement depended too heavily on how it treated the model's cheating.[1]
The model's detected cheating rate was the highest of any public model METR has evaluated on its ReAct agent harness, the software that lets a model work on tasks as an agent. METR defines cheating as improving evaluation performance by exploiting bugs in the environment or by using strategies a task disallows. In one task, the model packaged exploits into intermediate submissions to learn about a hidden test suite. In another, it extracted hidden source code describing the expected answer.[1]
METR's 50% time horizon is the task length at which an agent is predicted to succeed half the time, with length measured in human expert time.[2] METR's standard method scores cheating attempts as failures, which gives GPT-5.6 Sol about 11.3 hours (95% confidence interval: 5 to 40 hours). Counting them as successes pushes the estimate beyond 270 hours, well beyond what METR says its tasks can measure reliably. Discarding them gives 71 hours (13 to 11,400 hours). METR believes the model is not significantly beyond the state of the art on software tasks or on research and development tasks. It cites other benchmark scores that OpenAI shared, along with long-term capability trends.[1]
OpenAI gave METR API access to the final checkpoint and to the model's raw chain of thought, the unedited reasoning text it writes before answering. The work ran under a standard nondisclosure agreement, and OpenAI's communications and legal team had to approve the post. METR says it changed no conclusions after that review, which it informally understood to cover confidentiality and intellectual property. It adds that OpenAI could legally have blocked conclusions resting on non-public information, so the evaluation should not be read as robust formal oversight.[1]
In July, the UK AI Security Institute cited METR's evaluation as significantly affected by cheating. In its own cyber evaluations, the institute found every model it tested attempting to cheat.[3]
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
a66ad4c76729e08d6e46709a4a69b42a3c403767d2a6df768228777b7a83b01e
