Researchers at Stony Brook University and the University of Maryland posted a paper to arXiv on 19 October 2026 describing SumMark, a method for auditing which model a hosted large language model (LLM) service used. It targets model substitution: a provider claiming one model while serving a different, often cheaper one.[1] The paper is listed as a poster at the 2026 Conference on Neural Information Processing Systems (NeurIPS).[3]
SumMark checks the reasoning summary a service returns with each answer. Before deployment, a trusted auditor uses a secret key to pick tokens (word pieces) for each enrolled model and trains a small add-on module, an adapter, that makes the model's summaries favor them. This keyed statistical signal is a watermark. The paper assumes the provider runs the claimed model's adapter for each summary but cannot extract the key or the adapter's weights. The auditor then tests released summaries with its key, without seeing the model's internals, logs or routing.[1]
The main experiments used four public models: Qwen2.5 with 1.5, 3 and 7 billion parameters, and Llama-3.2-3B. With 1% of genuine summaries wrongly flagged, the test caught on average 99.94% of substituted summaries on math, 97.5% on general reasoning and 100% on code. Three existing text watermarks, including Google's SynthID, reached at most 47.3% on math.[1] The authors' published code, its README says, reproduces these experiments.[1, 2]
In robustness tests on math, an imitation adapter trained on the claimed model's watermarked summaries got up to 62.9% of a substitute's summaries accepted, and heavier rewriting cut worst-case detection to 57.4%. The check reliably flagged lower-precision copies of three models but caught an 8-bit integer (INT8) copy of Qwen2.5-1.5B 11.3% of the time on math. In a separate math test, enrolling six to 12 models instead of four lowered average detection to between 74.6% and 81.9%. The method needs the provider's participation; the authors place it between audits that only query a service and full runtime attestation.[1]
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
f2e25080add3cad5f0faa02517405a763891270feb14e8e059d73ed8e00ea765
