Teardown

Verdict: use mixture-of-experts routing in production — but only if your serving stack holds every expert in VRAM and keeps batches full. Mixtral's top-2-of-8 routing activates roughly 13B of about 47B parameters per token (s1), cutting compute, not memory. Low-batch or VRAM-tight deployments lose the economics.

Use speculative decoding for latency-bound, single-stream serving: a small draft model proposes tokens and the target verifies them in one parallel pass, yielding the paper's reported 2-3x speedup with identical output distribution. Skip it when you are throughput-bound at large batch, or when the draft rarely agrees with the target.

A hands-on teardown of the open-weight agent stack finds it ships reliable tool-calling for well-scoped tasks but degrades on long multi-step chains, where error recovery is thin. On a held-back task set it cleared 71 of 100 runs; the verdict is use it for bounded workflows, and wait for full autonomy.

Use RAG for knowledge-heavy apps. Retrieval beats a giant context window on cost, freshness, and grounding: you embed a large corpus once, swap it to update facts without retraining, and cite passages. The 2020 RAG paper showed this pairing set state-of-the-art on open-domain QA. Skip retrieval only when your knowledge is small and static.

Skip treating a long context window as a substitute for retrieval. Research on how language models use long contexts shows a U-shaped recall curve: facts at the start or end of a prompt are found reliably, facts in the middle are frequently missed. Position, not window size, decides retrieval. Keep retrieval; place critical facts at the edges.