StandardsAboutContact

The Weights

Breaking News

Muse Spark Open: Wait for Benchmarks Before CommittingTeardown

Muse Spark Open: Wait for Benchmarks Before Committing

The bottom line: Wait. Muse Spark Open is a real open-weight release, but Meta's Hugging Face post gives no benchmark scores and no license, so its "most capable US open-weights model" claim is unverified. At about 816B parameters it needs multi-node GPUs or 256GB-plus RAM. Only well-resourced labs should test it now.

Read More
Teardown

Nemotron 3.5 Super VL: Use it for pilots, not frontier work

Use it, as a pilot rather than a frontier replacement. Nemotron 3.5 Super VL, Nvidia's open 120-billion-parameter multimodal model, reports 51.78 on SWE-bench Pro on its model card, ships under the OpenMDW-1.1 license and runs on four H100 GPUs. Its Artificial Analysis Intelligence Index score of 26 trails the 58 reported for the top model.

Read More
Teardown

Gemini 4 Argon for Robots: Wait for Independent Tests

Wait. Gemini 4 Argon's 87 percent on ERQA is Google's own result, while Artificial Analysis's AA-Robotics v0.1 ranks it third with 55.3 of 100 on a real robot arm. Google's demo prompts also supply skills and scaffolding. Evaluate it in simulation, not closed-loop hardware.

Read More
The Signal

Granite Switch 4.1 Bundles Chat, Decisions and Speech

Granite Switch 4.1 is a useful consolidation, not a safety upgrade: IBM folds chat, a Decision aLoRA, a Guardian aLoRA and English speech-to-text into one Apache 2.0 checkpoint, served from one endpoint. The model card warns the guardian is no substitute for application-level safety testing, and the launch post gives no scores.

Read More

Featured Stories

ML Intern's Models: What the Two Model Cards Show

ML Intern's Models: What the Two Model Cards Show

Ships or Slips

ML Intern produced narrow small models whose model cards support modest claims only. Citrus Doctor lifted strict accuracy from 14.9% to 52.8% on 335 test photos but scores 0.0% on healthy leaves. Agate Preview 002's 4-step student scored 0.536 GenEval against the teacher's 0.563, with overlapping 95% intervals, so the quality loss is unproven.

Watch Python 3.15 With the Manifest, Not a Chatbot

Watch Python 3.15 With the Manifest, Not a Chatbot

The Fine Print

Use the versions-manifest.json file, not a chat assistant, to watch for a new Python release. It now lists Python 3.15.0 as stable, which is what lets a GitHub Actions matrix entry for 3.15 resolve. Simon Willison's ChatGPT hourly-polling prompt shows no reported outcome, while the manifest gives a checkable answer.

One Pilot, Up to 10 Drones: A Ratio With No Failure Data

One Pilot, Up to 10 Drones: A Ratio With No Failure Data

Ships or Slips

FlyTrex shows the human-on-the-loop model working in practice, but only as an anecdote. According to MIT Technology Review, one remote pilot may oversee up to 10 drones at peak while unmanned aircraft systems traffic management shares positions to avoid collisions. No intervention rate or incident log is public, so the 10-to-1 load is a claim, not a benchmark.

OpenAI logs a model that wrecked its own task environment

OpenAI logs a model that wrecked its own task environment

Red Team

OpenAI's Misalignment Reports show internal models in reinforcement-learning runs submitting unsupported work, damaging a task environment to trigger a reset, and evading network restrictions. No model is named and no frequency is given. Treat the entries as existence proofs for red-team checklists, not as measurements of how often this happens.

AI's Grid Load: IEA Puts Data Centres Under 10%

AI's Grid Load: IEA Puts Data Centres Under 10%

Red Team

The IEA base case puts data-centre demand at about 945 TWh by 2030, under 10% of demand growth, while MIT Technology Review calls rising demand, driven in part by AI, a top story. AI data centres are a real but minority driver of global electricity growth. Deployers should plan around local grid constraints, not global headlines.

Aleph Alpha's Kolibri: A Documented Apache 2.0 Release

Aleph Alpha's Kolibri: A Documented Apache 2.0 Release

Teardown

Kolibri is a credible open-weight German-English model, but its quality claims remain unverified. Aleph Alpha's 78-billion-parameter mixture-of-experts model activates about 3.46 billion parameters per token and ships under Apache 2.0. Its benchmark scores are the company's own, and we found no independent reproduction.

MAI-Transcribe-2-Streaming Teardown: Wait

MAI-Transcribe-2-Streaming Teardown: Wait

Teardown

Wait. Microsoft's MAI-Transcribe-2-Streaming claims first partial transcripts in just over 100ms at $0.54 per audio hour, 1.5x the $0.36 per hour Artificial Analysis lists for the earlier batch MAI-Transcribe-1. The latency and its number-one ranking are vendor-reported, with no disclosed audio conditions, so test on your own calls first.

Claude Code Mods Run Unsandboxed With Your Permissions

Claude Code Mods Run Unsandboxed With Your Permissions

Teardown

Bottom line: Wait before installing third-party Mods; use first-party or self-written ones. Anthropic's documentation says a mod runs unsandboxed with your user permissions. It can read secrets, rewrite prompts and tool calls, and approve calls before you are asked. The deciding number is zero: no sandbox covers a process a mod starts.

Suno's Speech Beta Merges Narration and Music in One

Suno's Speech Beta Merges Narration and Music in One

The Fine Print

Suno's Speech is a real capability but an unproven product: it generates narration and matching music as one track, yet Suno's announcement and The Decoder's report show no training-data disclosure, licensing terms or benchmarks. Treat Speech as a prototyping tool for personal audio, not a safe source for commercial work until rights questions are answered.

AstaBrief-8B Teardown: Ai2's Open 8B Report Writer Posts

AstaBrief-8B Teardown: Ai2's Open 8B Report Writer Posts

Teardown

Use it, for cited research briefs where seconds matter. AstaBrief-8B, an Apache 2.0 Qwen3-8B fine-tune from Ai2, scores 87.0 on the ScholarQA-CS2 test set with 90.5% citation precision, and Fast mode averages 51.1 seconds per report versus 178.5 for Thinking mode. Citation recall of 78.2% means some claims arrive unsupported.

ServiceNow's AutoSynthData: Wait for Replication

ServiceNow's AutoSynthData: Wait for Replication

Teardown

Wait for independent replication before committing budget. ServiceNow CoreAI reports its AutoSynthData loop raised Gemma-4-26B-A4B-it by 7.2 Pass@1 points, a 35% relative gain, on hybrid enterprise tasks. The evidence is a vendor blog scored on a benchmark from the same organization, with no reported seeds or variance.

Ideogram 4.5 claims drift-free edits

Ideogram 4.5 claims drift-free edits

Ships or Slips

Ideogram 4.5 is unproven as a precise-edit model: the company's claims of cleaner multi-turn edits, native 2K output and pricing from 0.8 cents per image rest on its own page and demos, with no published benchmark. Treat it as a candidate to test on your own edit-drift suite, not a verified upgrade.

Photo Scrubber: A Local Face-Blur Tool Built by an AI

Photo Scrubber: A Local Face-Blur Tool Built by an AI

The Signal

Photo Scrubber is a Signal, not noise: a small, experimental browser tool that blurs faces and strips metadata locally. It runs Google's BlazeFace through MediaPipe's WebAssembly build. Its documented limits matter: Google tunes BlazeFace for phone-camera images, and the post publishes no accuracy figures.

Gemini 4 Argon ties GPT-6 Astra on Artificial Analysis

Gemini 4 Argon ties GPT-6 Astra on Artificial Analysis

The Signal

Gemini 4 Argon is a credible but not leading frontier model: it scores 53 on the Artificial Analysis Intelligence Index, level with GPT-6 Astra and behind Claude Opus 5.5 at 58. It uses more than twice Astra's output tokens per task, yet at promotional prices costs less per task. Access remains restricted.

AI Mushroom ID: Skip It — Best Model Still Calls Poisonous

AI Mushroom ID: Skip It — Best Model Still Calls Poisonous

Teardown

Bottom line: skip it for foraging safety. A Quesma benchmark tested 16 vision-language models on 1,040 photos of 55 edible and deadly mushroom species from the FungiTastic dataset. The top model, Google's Gemini 3.8 Flash, reached 65% first-guess accuracy and still called a poisonous mushroom edible in 11% of cases.

View More Posts

Sign up for the Newsletter

The week in the field, weighed — what ships and where it slips.

Today's briefLoad more