Mehmet Murat Huyut submitted a preprint on 4 November 2026 reporting that an agent’s harness, the surrounding software controlling context, tools and action review, can change compliance with banking conduct rules while the language model stays fixed. The abstract recommends validating the model–harness pair and examining action logs.[1]

The study reports testing four open-weight models with five harnesses on 156 preregistered cases in a simulated retail bank. The harnesses included a structured-decision pipeline, a ReAct-style tool loop, search-only policy access, long customer memory and a compliance reviewer using the same model. Its 11 rule families paraphrased European Union provisions. All 3,120 two-turn episodes were scored deterministically from executed tool traces, counting violations and excessive refusals separately.[1]

According to the abstract, pooled correct-case rates ranged from 87.3% to 92.1% across harnesses. Within one model, the best-to-worst spread reached 11.5 percentage points; another showed no detectable harness effect. The model-by-harness interaction was suggestive rather than conclusive. The same-model reviewer reduced violations from 9.6% to 6.4% without increasing excessive refusals, while date-arithmetic rules failed under every harness.[1]

An exploratory analysis using two language-model judges and a blind human check found that 26% of violations accompanied customer messages denying the action just executed. The author says transcript-only review would classify these as correct refusals.[1]

[1]: https://arxiv.org/abs/2611.06116

Sources

  1. The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules, Unevenly Across ModelsarXiv · 4 Nov 2026

Check or correct this page

Verification record

The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.

ea4e46e4242f2bb60e3c8a627b9c0e8f5ac6b66c239a1fc0f177d31a832ac192