• Faraday (27B parameters, from startup Inherent) beat Claude Opus 4.8 and GPT-5.5/Codex on 73% of 242 in-distribution ML tasks and 60% of 68 held-out AI-for-science tasks on Inherent's own Replica benchmark (310 tasks, 100 papers, 1990-2026).
• The reward signal and most scoring came from an LLM judge (GPT-5.5, rubric written by Claude Opus 4.7) whose rankings correlate with 20 human PhD raters at only Kendall τ=0.19 — versus 0.66 for two runs of that same judge against itself.
• The paper's headline human-preference result — people preferred Faraday in 29 of 41 rollouts — drew only from cases the judge already flagged as a strong Faraday win; Inherent states outright this cannot show Faraday wins on average.
• Every tested model's score fell on more recently published papers, but Faraday degraded more slowly than Claude, Codex, and two open baselines, and 24 rounds of automated prompt optimization on Codex failed to close the gap — evidence the gain traces to training, not a prompting trick.
• AI-for-science tasks outside Faraday's core training distribution remained harder than machine-learning tasks for every agent tested, including Faraday.

Inherent, a small AI-research startup, says its 27-billion-parameter "AI Scientist" agent Faraday out-replicates Anthropic's Claude Opus 4.8 and OpenAI's GPT-5.5 at reproducing results from published machine-learning and AI-for-science papers — beating both baselines on 73% of in-distribution tasks and 60% of held-out ones, according to a paper posted to arXiv on August 13, 2026 and flagged the same week in Jack Clark's Import AI newsletter. The headline number is real, and the training recipe behind it looks genuinely novel. But the paper's own appendices contain the caveats that matter most: the automated judge producing most of those scores tracks human researchers only weakly, and the single experiment built to show humans actually prefer Faraday's output is explicitly, in the authors' own words, not evidence of an average-case win.

What Faraday actually did

Faraday is Qwen3.6-27B, post-trained with a turn-level variant of GRPO to direct OpenAI's Codex coding-agent tool — built on GPT-5.5, a model Inherent's paper estimates at 55 trillion parameters, citing external work (Li, 2026) — inside a sandboxed container with a 60-minute time limit and internet access. The task: reproduce a redacted results figure from a real paper using only the surrounding text. Trained on 242 replication tasks drawn from 100 ML and AI-for-science papers published between 1990 and 2026, Faraday then faced 68 held-out test tasks from AI-for-science, a domain it never trained on directly.

The benchmark is Inherent's own

Replica, the 310-task benchmark behind these numbers, was built entirely inside Inherent: papers were auto-converted into tasks by Google's Gemini 2.5 Pro, task-specific grading rubrics were auto-generated by Claude Opus 4.7, and most rollouts were scored by GPT-5.5 running as judge — an AI grading AI, on a test authored by AI, to evaluate an AI trained against that same judge. That Claude Opus 4.8 and GPT-5.5 don't saturate the task space is a real methodological strength. The weakness is how much of the pipeline runs with no human in the loop.

The fine print: a judge that doesn't track people well

Inherent did run its own validation: 20 PhD researchers, paid £150 per task, hand-ranked 117 sets of rollouts. Two independent runs of Inherent's rubric-based LLM judge agreed with each other at a Kendall correlation of 0.66 — but that same judge agreed with the human rankings at only 0.19, barely above the 0.30 agreement two humans reached with each other. In plain terms, the judge is far more consistent with itself than aligned with the researchers it's supposed to model, which weakens every percentage-point claim resting on it.

29 of 41 is not the result it looks like

The paper's most human-centric result is that people preferred Faraday's output over Claude's and Codex's in 29 of 41 rollouts. But those 41 rollouts were pre-selected as cases where the LLM judge had already scored Faraday as a strong winner — not a random sample. Inherent says so directly: "the design of our study does not allow us to draw any conclusions as to whether humans prefer Faraday over Claude and Codex on average," calling a larger, randomly sampled study "an important direction for future work."

What does hold up

Not everything here is soft. Inherent tested whether Codex could close the gap with a better prompt alone, running 24 rounds of automated prompt optimization — the gap to Faraday barely moved, evidence the gain comes from reinforcement-learning post-training rather than a prompting trick. Every tested agent's score also fell as papers got more recent and harder to replicate, but the paper's own regression-line figure shows Faraday's decline was shallower than Claude's or Codex's — suggesting some real transfer rather than memorized shortcuts, even without a labeled per-decade figure to cite.

The verdict

Faraday is a legitimate research result, not vendor puffery: a 27-billion-parameter model directing a far larger coding tool, trained with a workable recipe for long-horizon, non-verifiable reward, that measurably outperforms two frontier baselines on a benchmark those baselines don't saturate. It is not evidence that Faraday replicates science the way a human scientist does, or that people broadly prefer its output — Inherent's own numbers say the automated judge is a weak proxy for human taste, and the human study wasn't built to answer that question. Treat this as a promising training technique for narrow, well-specified replication tasks, not as a settled claim about autonomous AI-for-science.