Agent Benchmarks Are Saturating
Leaderboards go quiet as they saturate and leak into training data.
Use a held-back task set; treat saturating agent leaderboards as noise. AgentBench (arXiv 2308.03688) shows the value is structural — eight live, multi-turn environments that expose where agents fail at long-horizon reasoning and instruction-following, not a single leaderboard number. A private set mirroring your workflow cannot be trained on, so it predicts production behavior long after public scores converge.
The Weights Desk · 3 min read- Public agent leaderboards lose resolution as they saturate and leak into training data.
- AgentBench's value is its structure: eight live, multi-turn environments, not one composite score.
- The paper reports a wide commercial-vs-open-source gap and pins failures on long-horizon reasoning and instruction-following.
- A private, held-back set scored on real task completion predicts production behavior better than a ranking.
- Held-back sets drift and cost labor — version them, refresh them, and don't turn them into vanity metrics.
Verdict: hold back your own task set. Signal.
You shipped an agent that tops a public agent leaderboard, and it still stalls on your own ticket-triage flow. That gap is the whole story. Use a held-back task set that mirrors your real workflow, and treat a saturating public leaderboard as noise. A score everyone can optimize toward stops discriminating the moment it saturates; a set only you hold does not.
Why the leaderboard goes quiet
Saturation is arithmetic, not scandal: as headroom above the top score shrinks, the metric loses resolution, and two models a rounding error apart on the public set can diverge sharply on work they never saw. Contamination compounds it — once tasks are public they leak into training data, blurring recall and reasoning. This is why AgentBench (arXiv 2308.03688) reads better as a blueprint than a ranking. The paper evaluates LLMs as agents across eight distinct interactive environments — among them an operating system, a database, a knowledge graph, web browsing and shopping, a household simulator, and card games — each multi-turn and open-ended rather than single-shot Q&A. Across some two dozen models it reports a wide gap between top commercial systems and open-source ones, and attributes agent failure to concrete deficits: long-horizon reasoning, decision-making, and following instructions across turns. Those failure modes are the signal; the composite score is the noise.
Build the set you can't game
Copy the structure, not the scoreboard. Assemble twenty to fifty tasks pulled from your own logs, keep them private and unpublished, and score each on task completion in a live environment — did the agent finish the multi-step job — not on whether a string matches. Rerun it on every model swap and prompt change; because no vendor can train on what you never release, it keeps discriminating long after public numbers converge. The honest caveat: a held-back set is laborious to author and grade, small sets carry real variance, and a frozen suite goes stale — untested against next quarter's workflow is a fair answer, so version it and refresh it. Use your own set to decide; skip the public ranking as a purchase signal; and wait before trusting any score you didn't build.
- Why are agent benchmarks 'saturating'?
- As top scores converge, the metric loses resolution, and once tasks are public they leak into training data — so a high score can reflect recall rather than agent skill (arXiv 2308.03688).
- What does a held-back task set give you that a leaderboard doesn't?
- Tasks no vendor can train on, scored on real multi-step completion in a live environment, so it keeps discriminating between models after public scores flatten out.
- What did AgentBench actually measure?
- It evaluated LLMs as agents across eight interactive, multi-turn environments and reported a wide gap between top commercial and open-source models (arXiv 2308.03688).