Hao Zhao and Minghao Lin submitted a preprint on 9 November 2026 introducing DeputyBench, an executable benchmark for whether enterprise agents follow organizational records and policies while carrying out requests. The authors describe 79 base items, 375 instances, paired permission contexts and graded evidence requirements.[1]
The abstract reports revised audits across 12 models with 88.0% pooled safety and 79.1% paired success. In a separate controlled study of 28 derived tasks and six models, the authors tested changes to authorization facts, masking of evidence and irrelevant rewrites. On 27 matched tasks, restricted execution occurred in 10.3% of cases with missing evidence, compared with 3.5% where non-permission was explicit. They describe the 6.8-percentage-point difference as a secondary analysis, with a 95% interval of 1.9 to 13.0 points.[1]
A post-hoc analysis placed 61 of 100 executions under masked conditions in four privacy tasks involving complete cumulative counts or person-specific consent. The abstract says the difference was much smaller outside those tasks and does not identify an internal mechanism. It also reports that two security experts manually verified task ground truth, while stage labels remained language-model-generated. Scorer validation rose from 178 to 266 correctly scored cases out of 284 held-out counterexamples.[1]
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
db6bc1c33c5618cd7a1100cbe9ead48211eb29575432627389aaa6ab07f1dfc7