Ji Shi and seven coauthors submitted Trace2Eval to arXiv on 4 November 2026, proposing a dynamically updatable pipeline that builds coding-agent benchmarks from real-world interaction trajectories. The authors position it as a way to assess overall artifact quality in user workflows beyond correctness on static public tasks.[1]
According to the abstract, a multi-agent reconstruction system analyzes filtered trajectories, recovers the original user query, restores workspace files to their pre-interaction state, and rebuilds an execution environment intended to be equivalent to the recorded trajectory. A candidate agent then executes the reconstructed query to produce an initial artifact.[1]
The evaluation pipeline iteratively refines human rubrics through alignment among annotators and calibrates judge agents against human assessments using source-code and runtime evidence. The authors report that revising artifacts with pipeline feedback produced a mean score gain of 10.05 percentage points across 19 model-harness configurations. They report a maximum gain of 19.28 percentage points for DeepSeek-V4-Pro with Claude Code. These are reported evaluation outcomes; the saved abstract does not establish how well those gains transfer to other workflows.[1]
[1]: Trace2Eval: Contextualized Evaluation of Coding Agents from Real-World Traces. arXiv, 4 November 2026. https://arxiv.org/abs/2611.06506
Sources
Check or correct this page
Verification record
The digest below shows whether the text you are reading matches the published source. It does not show that the reporting is correct. The build computes it from this page's Markdown source and lists it in the site manifest.
4a6691f1a2042d0e50ebafce779d5aa5758c5f24f3fb2e7e311d7f148741e7f2