On 7 November 2026, the authors of Detent submitted a preprint describing a method for batch-invariant large language model (LLM) serving: a request should produce the same output bits regardless of which other requests share its batch. The abstract reports performance close to standard engine settings, with applications to reproducible evaluation, verifiable inference and reinforcement learning (RL).[1]

The authors identify kernel selection as a remaining source of variability even when individual kernels are batch-invariant. Engines choose kernels and their parameters using batch-dependent information; mathematically equivalent sums can round differently. Detent represents kernel selection as a program, identifies the smallest set of values that must remain fixed, and tunes those values for a workload before deployment.[1]

For multi-GPU Llama-3.1-70B and Qwen3-30B-A3B deployments in vLLM and SGLang, the abstract reports batch invariance. With tuning for each load, latency per output token was 1.04–1.17 times Stock, meaning each engine’s standard settings, compared with 1.67–10.3 times Stock for deterministic modes. It also reports bit-for-bit replay of seeded RL rollouts under different batches and cache splits, and matching trainer and rollout log probabilities at 1.03 times Stock’s rollout time per token. These are the authors’ reported results.[1]

[1]: https://arxiv.org/abs/2611.10611

Sources

  1. Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock SpeedarXiv · 7 Nov 2026

Check or correct this page

Verification record

The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.

f69479c94a9147c476bdb5d3e7dd9c65790d51af6768ff63319564d871ee593d