Taste-Bench: Best Agent Scores 59.7% on Strategic Forks
The best model passes a strict both-orders test on 59.7% of forks, and more reasoning effort barely helps.
Wait before trusting agents with strategic direction. On Taste-Bench, GPT-5.6 Sol scored 59.7% on a test that counts a fork only when the model picks the winning path in both option orders, against a 25% chance level. Accuracy fell to 21% when the deciding clue arrived late, and extra reasoning did not help.
The Weights Desk · 4 min read- Verdict: Wait. The top score on Taste-Bench is 59.7% (GPT-5.6 Sol), and the repository leaderboard lists GPT-5.5 at 59.5% and Claude Opus 5 at 55.5%.
- A question counts only if the model is right in both option orders, so chance is 25%. The 59.7% is a strict both-orders score, not a raw pick rate.
- Per The Decoder's report of the paper, accuracy averaged 62.3% when the deciding clue was already visible and 21.0% when it appeared only after extensive later work.
- Per the same report, moving two models from the lowest to the highest reasoning setting barely changed accuracy.
- In the authors' distillation test, advice from a Qwen3.6-27B student lifted success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%, against a 39.0% ceiling. The task set is small.
On Taste-Bench, the best model tested, GPT-5.6 Sol, scores 59.7% on a deliberately strict test of strategic judgment, and The Decoder reports that thinking longer does not help. The benchmark targets a gap that task-completion scores hide: whether an agent makes good intermediate choices while a long job unfolds. We did not run the harness ourselves. Leaderboard and scoring figures below come from the project repository; the rest come from The Decoder's report of the paper.
The bottom line: Wait
Verdict: Wait, for unsupervised strategic decisions. The one number that decides it is 59.7%, the best average on a scoring rule where chance is 25%. That is clearly better than guessing, yet a model that fails a large share of forks should not own direction on a long task. Use agents as executors with human-owned direction, and revisit the call if late-clue accuracy, reported at 21.0%, moves materially.
How the benchmark is built and scored
Taste-Bench asks one narrow question: given a trajectory up to a fork, which continuation succeeded? The repository README lists 502 questions, 390 engineering and 112 research. Forks are either parallel, where two independent attempts diverge, or detours, where an agent corrects itself within one run. The Decoder reports that the questions are mined from scored runs, including SWE-bench Pro and METR's MALT transcript release, with a generator model filtering trivial forks and human review confirming almost all labels.
Why 59.7% is above chance but still weak
The score is a both-orders score, so it is not a raw pick rate. Each question is asked twice with the options flipped, and it counts only if both answers are right. The README states that random guessing scores 25 and always choosing the same position scores 0. The README leaderboard puts GPT-5.5 at 59.5%, Claude Opus 5 at 55.5% and Grok 4.5 at 54.6%, with the 14-model field running down to 15.7%.
Accuracy collapses when the clue arrives late
Evidence timing explains much of the spread. The Decoder reports mean accuracy of 62.3% when the deciding clue was visible in earlier steps and 21.0% when it appeared only after extensive later work, which is below the 25% chance level. Detour forks were consistently harder than parallel ones. In production terms, this is the decision that looks fine at step 40 and only reveals itself at step 400.
More reasoning did not buy better judgment
Extra inference spend did not translate into better forks. The Decoder reports that two models, GPT-5.6 Sol and GPT-5.6 Luna, were tested at three reasoning budgets and that moving from the lowest to the highest setting barely changed accuracy. It also reports that models spent the most reasoning tokens on the hardest tier. The authors attribute the flat result to the needed clue not yet existing at those forks, which a larger thinking budget cannot supply.
Taste can be trained, within limits
The authors trained a student on Qwen3.6-27B from a teacher that knew the correct path, lifting judgment accuracy from 30.0% to 47.9%, per The Decoder. Used as advice to an agent on 41 held-out SWE-bench Pro tasks, success rose from 14.6% to 33.7%, against a 39.0% ceiling with perfect advice. Forty-one tasks is a small set, so this is a lead to reproduce, not a settled result.
What to do with this in production
Do not let an agent choose between architectures, rewrites or research directions without a checkpoint. Place human review at forks where the deciding evidence will arrive late, and score outcome-labelled decisions in your own traces. The repository provides a harness for this, and the dataset is gated on Hugging Face to limit training contamination. Treat any reported gain from distilled judgment as unproven until it replicates on your own tasks.
- What does Taste-Bench measure?
- It measures whether a model can choose the direction that later succeeded at a decision fork in a long-running agent task. Each of the 502 questions gives the task, the trajectory so far and two candidate directions.
- Is 59.7% close to a coin flip?
- No. The scoring rule requires a correct answer in both option orders, so random guessing scores 25% and always picking the same position scores 0, per the repository README. The best models are well above that floor but still fail a large share of forks.
- Does more reasoning effort fix the problem?
- Not in the authors' results as reported by The Decoder. Two models tested at three reasoning budgets barely changed accuracy, and the authors attribute this to the deciding clue not yet existing at the hardest forks.
- Can the evaluation be reproduced?
- Partly. The repository gives a command-line harness and says a full run is about 1,004 requests and roughly 8 million input tokens. The dataset is gated on Hugging Face and requires an access request.