Speculative Decoding: Where the Draft-Model Speedup Holds
A small draft model plus one parallel verification pass can cut latency 2-3x with identical outputs — but only when decoding is memory-bandwidth-bound…
Use speculative decoding for latency-bound, single-stream serving: a small draft model proposes tokens and the target verifies them in one parallel pass, yielding the paper's reported 2-3x speedup with identical output distribution. Skip it when you are throughput-bound at large batch, or when the draft rarely agrees with the target.
The Weights Desk · 3 min read- Speculative decoding trades cheap draft-model compute for fewer expensive target-model steps, and the target's output distribution is provably unchanged.
- The paper reports 2-3x wall-clock speedups over a standard T5X implementation, with no retraining and no change to outputs.
- The win comes from decoding being memory-bandwidth-bound: one parallel verify pass does the work of several sequential steps.
- It stops paying when the draft model is too costly relative to the target, or its token acceptance rate is low.
- Throughput-bound, high-batch serving is compute-bound, so the latency lever can shrink or go flat there.
Verdict: use it for latency, not throughput
If you serve one stream at a time and you care about time-to-token, turn speculative decoding on. It is one of the rare free lunches in inference: the paper 'Fast Inference from Transformers via Speculative Decoding' (arXiv, s1) reports 2x-3x wall-clock speedups over a standard T5X implementation with the outputs left exactly as they were — no retraining, no quality trade. The one thing that decides your result is whether your decode is memory-bandwidth-bound and whether you have a small draft model the target actually agrees with. Get both right and the speedup is real. Miss either and it evaporates.
How the trick works
Autoregressive decoding is sequential and starves the hardware: each step emits one token and waits. Speculative decoding breaks that by letting a cheap draft model guess several tokens ahead, then running the expensive target model once, in parallel, over all of them (s1). A modified rejection-sampling rule accepts the longest correct prefix and resamples the first miss. The math guarantees the kept tokens follow the target model's exact distribution (s1) — so this is a latency optimization, not an approximation of the big model. You are skipping its idle time, not its judgment.
Where it stops paying off
The gain is not unconditional. The paper's own analysis ties the speedup to two knobs: the draft model's acceptance rate and its cost relative to the target (s1). A draft that disagrees often, or one that is not much cheaper to run, shrinks the window fast. And the whole premise is that decoding wastes hardware — true at small batch, weaker at large batch, where the accelerator is already compute-bound and the extra draft-plus-verify work competes for the same cycles. That is the honest caveat: this is a single-stream latency lever, and it can go flat once you pack the batch for throughput. Measure acceptance on your actual traffic before you commit. If the draft rarely agrees with the target on your domain, skip it and spend the effort on a better draft.
- Does speculative decoding change the model's output quality?
- No. The paper's modified rejection-sampling scheme guarantees the kept tokens follow the same distribution as sampling from the target model alone, so quality is unchanged.
- What decides whether you actually get a speedup?
- Two things: how often the draft model's tokens are accepted by the target, and how cheap the draft is relative to the target. Low acceptance or an expensive draft erodes the gain.
- When should I skip it?
- When serving is throughput-bound at large batch sizes, where the hardware is already compute-bound and the extra draft-plus-verify work has little idle capacity to absorb.