• Bottom line: Wait. The retail gain is real on paper (Pass^4 57.5% to 67.5% with gpt-4.1), but it rests on 40 held-out tasks and IBM's own reporting.
• The deciding number is cost: retail cost per conversation went from $0.084 to $0.139 (+65%) with the full guideline set.
• The best configuration differs by domain. Retail did best with all 292 guidelines kept, while airline Pass^4 with gpt-4.1 was highest at the support-3 filter (45.0% on the test split).
• The gain shrinks but persists on a frontier model: gpt-5.5 airline Pass^4 went from 55.0% to 70.0% on 20 test tasks.
• Nothing here involves weight updates, so the method is cheap to try and cheap to test against your own task set before you trust it.

Bottom line: Wait. ALTK-Evolve, an IBM Research method for learned agent memory, raised gpt-4.1's retail Pass^4 on τ²-bench from 57.5% to 67.5% on held-out tasks, according to IBM Research's own write-up. The number that decides it is the price of that gain: cost per conversation rose 65%. Combined with test sets of only 40 retail and 20 airline tasks and no independent replication, the result is a strong lead to test, not a settled production recommendation.

What the method actually does

ALTK-Evolve runs an agent on training tasks, then uses an LLM to turn the successes and failures into short behavioral rules that are appended to the system prompt. In IBM's setup the same model, gpt-4.1, extracts the guidelines and acts on them, and no weights change and no human annotation is involved. The guidelines are extracted once from training tasks and applied unchanged to the held-out test set, which is the right shape for a fair comparison.

The retail and airline numbers

With gpt-4.1 on the held-out split, retail Pass^4 moved from 57.5% to 60.0% with a support-3 filter and to 67.5% with all 292 guidelines kept; mean accuracy went from 78.1% to 82.5%. Airline Pass^4 moved from 35.0% to 45.0% (support-3, 14 guidelines) and 40.0% with all 101 guidelines. Pass^4 requires success on all four runs of a task, so it measures reliability, which is what a customer-facing deployment needs.

The cost the headline leaves out

The full guideline set costs real money per call. Retail cost per conversation rose from $0.084 to $0.139 (+65%), and airline from $0.081 to $0.107 (+32%), because the guidelines add input tokens to every turn. Assistant turns fell slightly (retail 10.59 to 10.16), which offsets little of that. The authors say prompt caching could cut the added cost but did not measure it, so the real production figure is unknown.

The gain persists on a frontier model, with limits

On airline with gpt-5.5, baseline Pass^4 was 55.0%, rising to 70.0% with either the support-2 set (103 guidelines) or the full set (146). The effect is smaller in relative terms than on gpt-4.1 retail but still large. The sample is 20 tasks, so one task flipping moves Pass^4 by five points. That is too coarse to rank the two guideline configurations.

Why the leaderboard claim needs a footnote

IBM reports first place on the October 12, 2026 leaderboard snapshot: 67.5% retail Pass^4, ahead of a fine-tuned 30B model at 61.4%, and 82.0% airline with gpt-5.5 against 66-72% for frontier entries. The leaderboard uses the full base split, so these figures are not the same as the test-split table. They also come from the authors' own submissions, and the best guideline setting varies by domain, which hints at tuning to the benchmark.

The bottom line: Wait, then test it on your own tasks

Wait. The method is cheap to try because it needs no training, and the open-source repository makes a trial straightforward. But a 10-point Pass^4 gain on 40 tasks, bought with a 65% cost increase, is not enough to justify a rollout. Before adopting it, reproduce the gain on a held-out set from your own domain and report Pass^k alongside cost per resolved task. If the gain holds and caching shrinks the cost, upgrade the call to Use it.