StandardsAboutContact

The Weights

Breaking News

Meta's Muse Avatar Latency Claims Rest on Its Own TestsThe Fine Print

Meta's Muse Avatar Latency Claims Rest on Its Own Tests

Meta's own research post gives Muse Realtime Avatar numbers worth arguing with: about 870 ms latency to first byte and 12 concurrent sessions per GB200. But they are company-measured on one hardware stack, and Meta has not said the voice mode now reaching users runs on it.

Read More
The Signal

GDPx: no AI agent tops 12% on real-work tasks

No model clears 12 percent on SemiAnalysis' GDPx, and the benchmark is better read as a map of agent weaknesses than a ranking. Claude Opus 5.5 leads at 12.0 percent, with Gemini 4 Carbon at 10.9 percent and a third of the cost per task, but small per-domain samples limit precision.

Read More
Teardown

StateBench Verdict: Wait on Long-Document State Tracking

Wait before trusting frontier models with long-document state tracking. In Ai2's StateBench, Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna and Gemini 3.8 Flash handled simple updates, but accuracy on the hardest level fell to 13% or lower at the longest lengths. Gated DeltaNet models trained on the task did far better.

Read More
Teardown

Palette: IBM's Self-Hosted 20B Deck Model, Teardown

Bottom line: Wait. Palette, an IBM Research slide-deck system built on a self-hostable 20B open-weight model, scored 59.7 on the PresentBench three-domain mean against NotebookLM's 60.8. The result is close, but the weights, code and paper are not linked, and the evaluation still needs a frontier model to draft plans.

Read More

Featured Stories

Metis Reasoning: AlphaGo Idea, No Benchmarks Yet

Metis Reasoning: AlphaGo Idea, No Benchmarks Yet

Teardown

Wait. Metis Reasoning, a London startup founded by ex-DeepMind researcher Thore Graepel, proposes a knowledge ledger that separates facts from reasoning so conclusions can be checked against evidence. It has published no benchmarks and no working system. Until the company releases numbers, treat it as an architecture claim, not a hallucination fix.

Context Layer for Enterprise AI: Wait, the Evidence Is Thin

Context Layer for Enterprise AI: Wait, the Evidence Is Thin

Teardown

Wait. The context-layer pitch in this sponsored MIT Technology Review Insights report is a plausible diagnosis but not tested evidence: only 12% of surveyed organizations run AI agents in production, no sample size is disclosed, and the sponsor, Atlan, backs the proposed fix. Treat it as a hypothesis to pilot, not a buying signal.

Sponsored 'Industrial Intelligence' Piece Lacks

Sponsored 'Industrial Intelligence' Piece Lacks

The Signal

Treat this sponsored article as vendor positioning, not evidence. Its two case-study results are presented without baselines, sample sizes or methods, so they cannot be checked. MIT Technology Review labels the piece as produced by Microsoft; its governance checklist is useful, but its performance figures are unverified claims.

GPT-6.1 Terra: Half the Price, Real Gains, Open Questions

GPT-6.1 Terra: Half the Price, Real Gains, Open Questions

The Signal

GPT-6.1 Terra is a real price cut with measurable gains, but independence is limited. Artificial Analysis reports prices of $1 input and $5 output per million tokens versus $2 and $12 for GPT-5.6 Terra, and a 6-point Intelligence Index gain at max effort. Signal for agent workloads, pending your own tests.

TasteVal's 2027 self-improvement signal is thinner

TasteVal's 2027 self-improvement signal is thinner

The Fine Print

Treat 2027 as a hypothesis, not a finding. TasteVal reports Opus 5.5 at a 2.3x compute multiplier over human experts, but the 95% interval runs 1.15 to 4.37, the tasks are unreleased, and the trend covers under a year. Jack Clark's shift, per The Decoder, rests on that thin evidence.

Unreleased Anthropic Model Stops Attacks 67%: Wait on It

Unreleased Anthropic Model Stops Attacks 67%: Wait on It

Teardown

Bottom line: Wait. Anthropic's claim that an unreleased model is better aligned than Mythos 5.1 rests on one resampling study of one incident. The internal research model stopped attacks unprompted 67% of the time versus 10% for Opus 5.5, but Anthropic's own report also gives 62%, and outsiders cannot test the model.

Claude's 180M-line proofs: strong, but only for RISC-V

Claude's 180M-line proofs: strong, but only for RISC-V

The Fine Print

Anthropic's result is credible but narrow: Claude produced machine-checked Rocq proofs that 123 open-source libraries cannot crash on inputs under 64 MB, and found hundreds of bugs fuzzing missed. The guarantee covers single-threaded RISC-V binaries on musl libc, not the x86 or ARM builds most teams actually ship.

Google Policy Ties Personal AI Data to an Ads Exception

Google Policy Ties Personal AI Data to an Ads Exception

The Fine Print

Wait before enabling Google's personalized AI features on sensitive data. Google's privacy policy still says it does not show personalized ads based on Drive, Gmail or Google Photos content, but now excepts data a user chooses to make available to other Google services that support personalized ads. Whether Personal Intelligence triggers that exception is unconfirmed.

Canva Craft: Wait for Independent Tests Before Switching

Canva Craft: Wait for Independent Tests Before Switching

Teardown

Wait. Canva Craft, the first image model Canva Research trained in-house, ranks fourth on DesignArena by Canva's own account, and first in portraits and landscapes. That rank is the one number that matters, and no independent test has confirmed it. Compare Craft against your current model through the Canva API on your own prompts.

Microsoft's Diagnostic AI Wins a Test

Microsoft's Diagnostic AI Wins a Test

Teardown

Microsoft's claim that its diagnostic tool beats doctors is plausible but unproven for clinical use. MAI-DxO was reported to solve far more published NEJM case puzzles than physicians barred from colleagues and reference books, yet that benchmark is not a prospective patient trial, and Microsoft AI has not shown real-world superiority.

Context-aware AI: a Celonis-sponsored pitch, not evidence

Context-aware AI: a Celonis-sponsored pitch, not evidence

The Signal

This 'case for context-aware AI' is a sponsored Celonis interview, not independent research. MIT Technology Review Insights issued it with no statistics or study, so the claim that agents need operational context is plausible but unmeasured. Treat it as a vendor pitch for the Celonis context model and demand a baseline evaluation.

Scalable AI Foundations: A Session With Little Evidence

Scalable AI Foundations: A Session With Little Evidence

The Signal

Noise, for now: MIT Technology Review's EmTech Future session on scalable AI names the right ingredients (data, engineering, platforms, operating models, governance) but, in the published summary, offers no benchmark, incident or named deployment to test. Treat it as a checklist, not evidence, until the full session shows measurable results.

VocabX's 77% Cost Cut Rests on One Grader

VocabX's 77% Cost Cut Rests on One Grader

Teardown

Wait. VocabX, Dharma-AI's vocabulary rebuild, cut measured inference cost 77% on one Qwen3-4B essay grader, and its bench score rose from 0.703 to 0.810. But the paper is unpublished, the bench is private, and no code or model is linked, so the result is a claim a team cannot yet reproduce.

Google Cloud's OpenAI Preview: Real

Google Cloud's OpenAI Preview: Real

The Fine Print

Google Cloud's OpenAI offering is confirmed but narrow. Google says GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna are in private preview for select enterprise customers in Model Garden, billed through Marketplace. No prices, regions or benchmarks are published, and zero data retention is still "coming soon." Treat it as binding soon, not binding now.

An AI kill switch is a testable control, not a moonshot

An AI kill switch is a testable control, not a moonshot

The Signal

An AI kill switch is a management control, not a moonshot. The NIST AI RMF Playbook already expects organizations to assign responsibility and mechanisms to deactivate AI systems that behave inconsistently with intended use. MIT Technology Review's letter groups an AI kill switch with fusion as ambition, but neither source measures how reliably shutdown works.

View More Posts

Sign up for the Newsletter

The week in the field, weighed — what ships and where it slips.

Today's briefLoad more