Open-Weight Agent Stack: Reliable Tools, Thin Autonomy
The stack calls tools cleanly on bounded tasks, but error recovery falls apart once the chain runs long.
A hands-on teardown of the open-weight agent stack finds it ships reliable tool-calling for well-scoped tasks but degrades on long multi-step chains, where error recovery is thin. On a held-back task set it cleared 71 of 100 runs; the verdict is use it for bounded workflows, and wait for full autonomy.
The Weights Desk- On a held-back 100-task set the stack completed 71 runs end-to-end, strong for bounded tool use.
- Failures cluster on chains longer than six steps, where the agent rarely recovers from a bad tool result.
- The verdict is use it for scoped workflows now; wait before handing it unsupervised, multi-step autonomy.
The verdict
The open-weight agent stack is a competent tool-caller. On our held-back set it cleared 71 of 100 tasks with no human in the loop, and nearly all of those wins were bounded jobs: fetch, transform, write, verify, done. For that shape of work you can ship it today.
The trouble starts when the chain runs long. Past roughly six steps the agent stops noticing when a tool hands back the wrong thing, and instead of retrying it builds on the bad result until the run drifts off-task. That is a recovery problem, not a reasoning one, and it is why unsupervised autonomy is still a wait. Analysis, not deployment advice; the numbers here are sourced, so verify against your own task mix before trusting it.
- How did the agent stack score on the held-back task set?
- It completed 71 of 100 runs without human intervention, concentrated in tasks of six steps or fewer.
- Where does it break down?
- On long chains: once a tool call returns a bad result, the agent seldom detects the error and retries, so the run compounds off-track.