TasteVal's 2027 self-improvement signal is thinner
A new benchmark shows a frontier model beating expert baselines at experiment design. The case for a 2027 timeline is narrower than the headlines.
Treat 2027 as a hypothesis, not a finding. TasteVal reports Opus 5.5 at a 2.3x compute multiplier over human experts, but the 95% interval runs 1.15 to 4.37, the tasks are unreleased, and the trend covers under a year. Jack Clark's shift, per The Decoder, rests on that thin evidence.
The Weights Desk · 4 min read- TasteVal scores models on 8 open-ended AI R&D tasks by how much compute they need to match human experts, using 24 experts and 20 models from 2023 to 2026.
- The top model, Opus 5.5, is reported at a 2.3x compute multiplier with a 95% interval of 1.15 to 4.37, so the lower bound barely clears parity.
- The reported 3.0-month doubling time (95% interval 1.7 to 5.0) covers only the period since December 2025. The same abstract says final normalized performance shows no trend break, doubling every 14.6 months.
- The tasks are withheld to keep the benchmark uncontaminated, so outsiders cannot rerun it.
- The usable finding for practitioners is narrow: models are getting better and cheaper at experiment design. The benchmark does not show unsupervised self-improvement.
A new benchmark reports that frontier models are improving quickly at designing and judging experiments, and The Decoder reports that Anthropic co-founder Jack Clark now sees self-improving AI as possible as early as 2027. The headline number is real but narrow: one benchmark, eight tasks, wide error bars and a trend window of under a year. It is a signal worth tracking, not a binding forecast, and it changes nothing a team shipping AI must do today. Method note: we read The Decoder's report and the paper's arXiv abstract, not the full 38-page paper.
What TasteVal actually tests
TasteVal tests experimental judgment, not general intelligence. The arXiv abstract defines research taste as picking problems, designing experiments and interpreting results, and measures only the experimental part. It covers 8 tasks, 24 human experts and 20 models released between 2023 and 2026. Score is compute efficiency: reaching an expert's result with half the compute counts as twice the taste. That is a measurable proxy for research skill, but it remains a proxy built on a small sample of tasks.
The headline result and its error bars
The top model beats the expert baseline, but the data does not pin down by how much. The abstract reports that Opus 5.5 reaches a 2.3x compute multiplier with a 95% confidence interval of 1.15 to 4.37, at about one-thirtieth of the baseliners' per-run cost. The lower bound sits barely above parity. The Decoder adds that the baseline is the best expert attempt per task. Read it as probably better than the human baseline on these tasks, not more than twice as good.
Why the trend line is the weak point
The doubling claim carries the forecast and is the least secure number. The abstract reports the multiplier doubling about every 3.0 months since December 2025, with a 95% interval of 1.7 to 5.0 months, versus roughly every 14 months from 2023 to December 2025. The same abstract says final normalized performance shows no trend break, doubling every 14.6 months. A few months of data, a changed slope and an interval nearly threefold wide make calendar extrapolation a judgment, not a measurement.
What we could and could not verify
We could match The Decoder's benchmark figures to the arXiv abstract, but we could not rerun or audit the results. The abstract says the tasks are withheld to keep the benchmark uncontaminated, which protects its validity but leaves readers unable to reproduce it. The Decoder credits P-Zero Research as the lab. The arXiv listing names two authors and no lab. Clark's forecast, including his earlier 2028 expectation, is The Decoder's account, not something we confirmed against his newsletter.
Verdict: a signal to track, not a deadline to plan around
Do not rebuild a governance or security plan around a 2027 date. TasteVal supports a narrower claim: on eight withheld tasks, the best model likely matches or exceeds expert experimental taste at a fraction of the cost. That matters for teams automating ablation and evaluation work, who should test it on their own tasks. It does not show unsupervised self-improvement. Revisit when an independent replication, more tasks or tighter intervals appear.
- What does TasteVal measure?
- It measures the experimental part of research taste: designing experiments and interpreting results. It does not measure choosing which problems to pursue. Score is compute efficiency, so matching an expert's result with half the compute counts as twice the taste. The arXiv abstract describes 8 tasks, 24 human experts and 20 models.
- Does TasteVal show AI can already improve itself?
- No. It tests experiment design and interpretation on 8 tasks against human baselines. It does not show a model running an unsupervised research loop that raises its own capabilities. Its own reported intervals are wide.
- Did Jack Clark move his forecast to 2027?
- According to The Decoder, yes. He now thinks recursive self-improvement could become possible as early as 2027, against an earlier expectation of 2028. We did not read his newsletter issue ourselves, so we report the claim as the outlet states it.
- Anthropic co-founder Jack Clark moves his prediction for self-improving AI forward to 2027 — The Decoder
- TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts — arXiv (Oliver Jaffe, Dane Sherburn)