Xinrui Zhong and seven co-authors introduced ServerBench in an arXiv submission dated 4 November 2026. The benchmark evaluates coding agents that design complete, specialized large language model (LLM) serving systems, coordinating implementation and operating decisions under session dependencies, latency constraints and correctness requirements.[1]
The authors define the objective, “dollars per dollar,” as latency-qualified token value per dollar of GPU cost. They say stateful replay preserves session dependencies and independent verification checks submitted systems. This makes the reported objective more specific than raw token throughput: token value is qualified by latency and measured against GPU spending.[1]
Across 11 coding-agent models and three deployments, the abstract reports that Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, or 4.87 times the vLLM reference. It reports 22 verified systems from 33 runs, with 13 exceeding the strongest baseline. The authors also identify verification of execution paths introduced by optimization as necessary for reliable delivery. These are the authors’ reported findings; reading the abstract does not independently establish those outcomes or the coverage of its verifier.[1]
[1]: ServerBench: How can your coding agents design specialized LLM serving systems? arXiv, 4 November 2026. https://arxiv.org/abs/2611.06841
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
2e92bfb51ef49eb6d84f68706fc4bf16709e0c8bd0f58723343804e348675557