StandardsAboutContact

The Weights

Breaking News

StateBench Verdict: Wait on Long-Document State TrackingTeardown

StateBench Verdict: Wait on Long-Document State Tracking

Wait before trusting frontier models with long-document state tracking. In Ai2's StateBench, Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna and Gemini 3.8 Flash handled simple updates, but accuracy on the hardest level fell to 13% or lower at the longest lengths. Gated DeltaNet models trained on the task did far better.

Read More
Teardown

Palette: IBM's Self-Hosted 20B Deck Model, Teardown

Bottom line: Wait. Palette, an IBM Research slide-deck system built on a self-hostable 20B open-weight model, scored 59.7 on the PresentBench three-domain mean against NotebookLM's 60.8. The result is close, but the weights, code and paper are not linked, and the evaluation still needs a frontier model to draft plans.

Read More
The Signal

Message Threads: 35% Median TTFT Cut Is Claude Code Data

Message Threads, Anthropic's Messages API beta, cut median time to first token about 35 percent for 500,000 to one million token requests, by Anthropic's own account. The figure comes from Claude Code production usage, not a controlled benchmark. Token usage and billing stay unchanged, and threads are best-effort, so teams must handle expiry.

Read More
Red Team

SETI's Telescope Agents Work, but Prove Little So Far

The Allen Telescope Array's agentic AI setup is a credible engineering upgrade but not yet a verified science result. The Crab pulsar detection shows the pipeline works on a known natural source, and the weeks-to-hours speedup comes from NVIDIA's account. No extraterrestrial signal has been found, and we found no independent paper on the agents.

Read More

Featured Stories

GPT-6.1 Terra: Half the Price, Real Gains, Open Questions

GPT-6.1 Terra: Half the Price, Real Gains, Open Questions

The Signal

GPT-6.1 Terra is a real price cut with measurable gains, but independence is limited. Artificial Analysis reports prices of $1 input and $5 output per million tokens versus $2 and $12 for GPT-5.6 Terra, and a 6-point Intelligence Index gain at max effort. Signal for agent workloads, pending your own tests.

Sponsored 'Industrial Intelligence' Piece Lacks

Sponsored 'Industrial Intelligence' Piece Lacks

The Signal

Treat this sponsored article as vendor positioning, not evidence. Its two case-study results are presented without baselines, sample sizes or methods, so they cannot be checked. MIT Technology Review labels the piece as produced by Microsoft; its governance checklist is useful, but its performance figures are unverified claims.

TasteVal's 2027 self-improvement signal is thinner

TasteVal's 2027 self-improvement signal is thinner

The Fine Print

Treat 2027 as a hypothesis, not a finding. TasteVal reports Opus 5.5 at a 2.3x compute multiplier over human experts, but the 95% interval runs 1.15 to 4.37, the tasks are unreleased, and the trend covers under a year. Jack Clark's shift, per The Decoder, rests on that thin evidence.

Unreleased Anthropic Model Stops Attacks 67%: Wait on It

Unreleased Anthropic Model Stops Attacks 67%: Wait on It

Teardown

Bottom line: Wait. Anthropic's claim that an unreleased model is better aligned than Mythos 5.1 rests on one resampling study of one incident. The internal research model stopped attacks unprompted 67% of the time versus 10% for Opus 5.5, but Anthropic's own report also gives 62%, and outsiders cannot test the model.

Claude's 180M-line proofs: strong, but only for RISC-V

Claude's 180M-line proofs: strong, but only for RISC-V

The Fine Print

Anthropic's result is credible but narrow: Claude produced machine-checked Rocq proofs that 123 open-source libraries cannot crash on inputs under 64 MB, and found hundreds of bugs fuzzing missed. The guarantee covers single-threaded RISC-V binaries on musl libc, not the x86 or ARM builds most teams actually ship.

Google Policy Ties Personal AI Data to an Ads Exception

Google Policy Ties Personal AI Data to an Ads Exception

The Fine Print

Wait before enabling Google's personalized AI features on sensitive data. Google's privacy policy still says it does not show personalized ads based on Drive, Gmail or Google Photos content, but now excepts data a user chooses to make available to other Google services that support personalized ads. Whether Personal Intelligence triggers that exception is unconfirmed.

Canva Craft: Wait for Independent Tests Before Switching

Canva Craft: Wait for Independent Tests Before Switching

Teardown

Wait. Canva Craft, the first image model Canva Research trained in-house, ranks fourth on DesignArena by Canva's own account, and first in portraits and landscapes. That rank is the one number that matters, and no independent test has confirmed it. Compare Craft against your current model through the Canva API on your own prompts.

Microsoft's Diagnostic AI Wins a Test

Microsoft's Diagnostic AI Wins a Test

Teardown

Microsoft's claim that its diagnostic tool beats doctors is plausible but unproven for clinical use. MAI-DxO was reported to solve far more published NEJM case puzzles than physicians barred from colleagues and reference books, yet that benchmark is not a prospective patient trial, and Microsoft AI has not shown real-world superiority.

Context-aware AI: a Celonis-sponsored pitch, not evidence

Context-aware AI: a Celonis-sponsored pitch, not evidence

The Signal

This 'case for context-aware AI' is a sponsored Celonis interview, not independent research. MIT Technology Review Insights issued it with no statistics or study, so the claim that agents need operational context is plausible but unmeasured. Treat it as a vendor pitch for the Celonis context model and demand a baseline evaluation.

Scalable AI Foundations: A Session With Little Evidence

Scalable AI Foundations: A Session With Little Evidence

The Signal

Noise, for now: MIT Technology Review's EmTech Future session on scalable AI names the right ingredients (data, engineering, platforms, operating models, governance) but, in the published summary, offers no benchmark, incident or named deployment to test. Treat it as a checklist, not evidence, until the full session shows measurable results.

VocabX's 77% Cost Cut Rests on One Grader

VocabX's 77% Cost Cut Rests on One Grader

Teardown

Wait. VocabX, Dharma-AI's vocabulary rebuild, cut measured inference cost 77% on one Qwen3-4B essay grader, and its bench score rose from 0.703 to 0.810. But the paper is unpublished, the bench is private, and no code or model is linked, so the result is a claim a team cannot yet reproduce.

Google Cloud's OpenAI Preview: Real

Google Cloud's OpenAI Preview: Real

The Fine Print

Google Cloud's OpenAI offering is confirmed but narrow. Google says GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna are in private preview for select enterprise customers in Model Garden, billed through Marketplace. No prices, regions or benchmarks are published, and zero data retention is still "coming soon." Treat it as binding soon, not binding now.

An AI kill switch is a testable control, not a moonshot

An AI kill switch is a testable control, not a moonshot

The Signal

An AI kill switch is a management control, not a moonshot. The NIST AI RMF Playbook already expects organizations to assign responsibility and mechanisms to deactivate AI systems that behave inconsistently with intended use. MIT Technology Review's letter groups an AI kill switch with fusion as ambition, but neither source measures how reliably shutdown works.

Stroke-Rehab Robot: Feasibility Shown, Recovery Unproven

Stroke-Rehab Robot: Feasibility Shown, Recovery Unproven

Teardown

The stroke-rehabilitation robot built by Johannes Lachner and Noah Geiger has demonstrated feasibility, not recovery. Its diffusion-based impedance learning approach is documented in a preprint, but the reported tests used healthy participants, and no stroke-patient outcomes are reported. Hospitals should treat it as research to watch until the Munich clinical study reports.

DeepInfer teardown: Wait on DeepSeek's inference engine

DeepInfer teardown: Wait on DeepSeek's inference engine

Teardown

Wait. DeepInfer, the Apache-2.0 inference framework DeepSeek open-sourced for DeepSeek-V4.1-Flash, is worth reading but not yet worth deploying: the release covers offline inference only, targets NVIDIA SM100 and Huawei Ascend 950 hardware, and omits the KV cache, disaggregation and load-balancing pieces a complete service needs.

View More Posts

Sign up for the Newsletter

The week in the field, weighed — what ships and where it slips.

Today's briefLoad more