StandardsAboutContact
The Weights
Reflection's Beam: Wait for Independent Evals

Reflection's Beam: Wait for Independent Evals

Beam is a 501B open-weight model with strong self-reported coding scores. Every number comes from its maker, and the hardware bill is steep.

The bottom line: Wait. Beam-1.0-501B-A23B, an Apache 2.0 mixture-of-experts model from Reflection, reports 80.7% on SWE-bench Verified at NVFP4, but every figure is vendor-supplied and none is independently reproduced. Running it sensibly needs four B200 GPUs, so hold off until third-party evals and real cost-per-call numbers exist.

The Weights Desk · 4 min read

The bottom line is Wait. Reflection's Beam-1.0-501B-A23B arrived on Hugging Face on October 16, 2026 with an Apache 2.0 license, 1M-token context and a benchmark table that looks competitive for coding agents. The table is the vendor's own. The Hugging Face post links no paper, and a pre-release announcement reported by VKTR said Reflection concedes Kimi K3 still leads on raw capability. A capability is a claim until it survives a test you can describe, and none of these has yet.

What the architecture buys you

Beam's design targets serving cost at long context. Per the Hugging Face post, it has 52 decoder layers, 256 experts per MoE layer with 8 routed per token, and sliding-window attention over 512 tokens in 43 layers, leaving 9 global layers. That cuts the long-context cache to about 36.08 GiB per 1M-token sequence, against 208 GiB if every layer were global. That is a memory figure, not a latency or cost-per-call figure.

The reported scores

Reflection reports 81.6% on SWE-bench Verified in BF16, 66.1% on SWE Bench Pro v1, 80.1% on Terminal Bench v2.1 and 90.6% on GPQA Diamond. The pre-release announcement, as summarized by VKTR and others, cited 80.9 on SWE-bench Verified, so the headline number already moved between documents. Nothing here states the harness settings, sampling budget or reasoning-effort level used for each score, so a reader cannot yet reproduce them.

Quantization holds up on paper

The FP8 and NVFP4 checkpoints lose little in Reflection's table. SWE-bench Verified is 81.6% in BF16, 80.2% in FP8 and 80.7% in NVFP4, while SWE Bench Pro v1 moves from 66.1% to 65.6% in NVFP4. HLE without tools falls from 36.0% to 34.8%. Differences this small can sit inside run-to-run noise, and the post does not report variance or repeated runs.

The hardware bill

Owning the model means owning a multi-GPU node. The NVFP4 checkpoint is about 280 GB, and Reflection recommends four B200s (720 GB total). Two B200s can hold the weights with little room left for cache. Older GPUs such as the H200 need the FP8 checkpoint, and the BF16 example spreads about 1 TB across 16 H200s. Day-0 support in vLLM, SGLang and transformers lowers integration risk, not the hardware cost.

Fine-tuning claims are narrow

The post's customization results are small demonstrations. LoRA SFT on Multilingual-Thinking made Beam reason in the requested language on 16 of 16 held-out prompts, versus 3 of 16 for the base model. GRPO with LoRA raised AIME25 accuracy from 42.9% to 64.2% while mean output length fell from 3,188 to 2,197 tokens. A 16-prompt test is too small to predict how tuning will behave on production data.

The bottom line

The bottom line: Wait. The one number that decides it is 80.7% on SWE-bench Verified at NVFP4: strong if it holds, but it is reported by the vendor, with no linked paper and no independent run. Teams with spare Blackwell capacity can test it on their own repositories now. Everyone else should wait for third-party evaluations and measured latency and cost per call before committing a four-GPU node.

Should a team adopt Beam now?
Wait. The scores are self-reported, there is no linked paper in the Hugging Face post, and no cost-per-call or latency figures for production traffic have been published. Run your own eval on your own tasks first.
What hardware does Beam need?
Per the Hugging Face post, four B200 GPUs (720 GB) are recommended for the NVFP4 checkpoint of about 280 GB. Two B200s can hold the weights with little room left. The BF16 checkpoint is about 1 TB, and the example given spreads it across 16 H200s.
How much does quantization hurt Beam's scores?
On Reflection's numbers, little. SWE-bench Verified is 81.6% in BF16, 80.2% in FP8 and 80.7% in NVFP4. HLE without tools drops from 36.0% to 34.8% in NVFP4. These are single reported runs, not a reproduced comparison.
  1. Welcome Beam: a 501B open model you can own, customize, and scale — Hugging Face
  2. Beam-1.0-501B-A23B model card — Reflection (Hugging Face)
  3. Reflection AI unveils Beam, a 501B open-weight model — VKTR