On 13 November 2026, Anthropic published recommendations for building AI safety evaluations, arguing that a model safety claim depends on how well the evaluation measures the behavior in question. The company says its evaluations inform deployment and safeguard decisions, and that results can change with details such as question format and answer order.[1]

Anthropic describes multi-turn evaluations in which a domain expert specifies a simulated user's objectives and tactics, a separate Claude model generates subsequent user turns, and a grader model scores the transcript against a risk-specific rubric. It says a conversation fails its appropriate-response measure if any turn is judged inappropriate. The company also recommends testing harmful and benign requests in the same domain, to measure both harmful compliance and over-refusal.[1]

The company says evaluation settings should match the intended deployment, including tool access and system prompts. It recommends checking graders against human labels, testing their sensitivity to changes in models or rubrics, and flagging grading failures. Anthropic also recommends running safety evaluations in a sandbox without internet access and monitoring them in real time; it says it has temporarily stopped many evaluations that give models internet access following its own incidents.[1]

Sources

  1. Building strong safety evaluationswww.anthropic.com · 13 Nov 2026

Check or correct this page

Verification record

The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.

2ffb61c450afdf904f9104961b735bb7f65162b84babc151794728e116249445