Speculative decoding

Draft several tokens. Let the large model check them together.

A language model usually generates one token at a time. A small model can propose several tokens cheaply, then a larger model can verify that proposal in one pass. When enough of the draft survives, the same large model produces its answer in less time.

6 sections · 5 figures · about 15 minutes · no model downloads

Why the next token makes you wait

An autoregressive language model predicts a distribution over the next token from the text so far. A token is a piece of text: it might be a word, part of a word, or punctuation. After selecting a token, ordinary decoding feeds it back into the model before choosing the next one.

Call the model whose output we want the target. For a single stream of text, its accelerator can spend much of a decoding step moving model weights from memory. Scoring a short block of known tokens can then take less time than generating those tokens in separate passes.This opportunity depends on hardware, model size, batch size and block length. The original Leviathan et al. paper (2023) analyzes the regime with enough parallel compute to verify a short block cheaply.

Ordinary decoding one target pass per token

target → 1target → 2target → 3target → 4

Speculative decoding draft first, then verify

1234target checks the block

Illustrative elapsed time →

A simplified schedule, not a benchmark. The draft model (orange) still runs sequentially. The target (blue) checks known tokens together. The saving comes from committing several tokens per expensive target pass.

Speculative decoding uses this asymmetry. A cheaper draft model guesses a continuation; the target decides how much to keep. The draft changes the work schedule, while the verification rule preserves the target’s decoding behavior.

Keep the prefix up to the first mismatch

Start with greedy decoding: each model chooses its most likely next token. The draft proposes four tokens. The target checks each choice, and we keep the consecutive matches from the beginning. At the first mismatch, we insert the target’s choice and discard the rest of the draft.

The discarded tokens were predicted after text we did not keep. Even a later match cannot rescue that suffix. Try the three cases below, then replay one round.These are hand-authored toy model outputs, with words treated as tokens. “Accepted” means it passes the decoding rule; it does not mean a statement is factually true.

Draft outcome Step 6 of 6

Committed prefixWe walked

Scroll sideways to see all five positions.

Output after this round

3 new tokens committed

Simplified greedy example. A round adds between one and five tokens when the draft length is four. If every draft token matches, the same target pass also supplies one bonus token. With reduced motion enabled, Play shows the finished round; Step remains available.

After a rejection, drafting starts again from the repaired prefix. After full acceptance, drafting starts after the bonus token. With the same tie-breaking and equivalent numerical results, this greedy procedure produces the same token sequence as ordinary target-only greedy decoding.

Why verification fits in one pass

Generating an unknown continuation requires sequential choices. Evaluating a known continuation is different: all the input tokens are already available. A causal transformer can score those positions together, with an attention mask that prevents any position from reading future tokens.

Each target prediction has its own prefix. For a four-token draft, verification gives four distributions to check the proposals and one more distribution for the possible bonus token.This is the linear draft–target algorithm in Chen et al. (2023), Algorithm 2. Cached-prefix handling and whether a boundary distribution was already computed are implementation details.

Scroll sideways to compare prefixes and decisions.

Target distribution	Text it can use	Checks / supplies	After rejection at token 3
p₁	We walked	along	Keep “along”
p₂	We walked along	the	Keep “the”
p₃	We walked along the	lake	Replace with “river”
p₄	We walked along the ~~lake~~	.	Discard: wrong prefix
p₅	We walked along the ~~lake .~~	possible bonus	Discard: wrong prefix

All five target distributions can be computed in one causal forward pass over the candidate block. Parallel scoring does not remove the dependence on the preceding text. This table uses the third-token mismatch from Figure 2.

Verification still performs work for tokens that may be thrown away. It pays off when processing the block is cheap enough and the accepted prefix is long enough. One pass does not mean one token’s worth of arithmetic.

Sampling needs a probability correction

Greedy agreement is only one case. When generating varied text, the target samples a token from a probability distribution. Let p be that target distribution and q the draft distribution at the same prefix. Draw a candidate x from q, then accept it with probability:

accept(x) = min(1, p(x) / q(x))

If the draft proposes a token too often, the ratio trims the excess. A candidate drawn from q has positive q(x), so its acceptance ratio is defined. On rejection, sample a replacement from the target’s missing probability mass:

correction(x) ∝ max(0, p(x) − q(x))

The ∝ symbol means “proportional to”: divide these values by their sum to make a distribution. Sampling from the unmodified target distribution after rejection would generally give the wrong result.The proof appears in Leviathan et al., Appendix A.1. Use the actual sampling distributions after temperature, top-k or top-p transformations. This page uses p for target and q for draft; Chen et al. use the opposite notation.

Draft probability of “lake” 50%

Probability of accepting a draft: 70%.

Scroll sideways to compare all probability columns.

Token	Target p	Draft q	Accepted mass	Correction¹	Final output

¹ Correction is a distribution conditional on rejection. Final output = accepted mass + probability of rejection × correction.

An exact probability calculation for a simplified three-token vocabulary, not a random simulation. The slider divides the remaining draft mass between river and sea in a 3:2 ratio; “Draft equals target” sets all three probabilities exactly. Changing the draft changes acceptance, but the final output distribution stays equal to the target.

At the default settings, “lake” is proposed 50% of the time but should appear only 20% of the time. It is accepted with probability 20 / 50 = 40%. The resulting accepted mass is 20%. Rejection happens 30% of the time overall; its correction goes to “river” and “sea.”

Apply this rule from left to right through the draft. At the first rejection, sample the correction and stop that round. If all drafts pass, sample a bonus token from the target’s next distribution. The resulting sequence has the target’s sampling distribution, subject to numerical precision. It need not be the same sequence produced by a target-only run with the same random seed.

When more guessing stops helping

A longer draft offers more possible tokens per target pass, but also takes longer to generate. Its later tokens only survive when all the earlier ones survive. Let α be the probability of acceptance at each position, assumed constant and independent in this simplified model.

With k draft tokens, a round always commits at least one token: either a replacement or the first accepted token. It commits a second token with probability α, a third with probability α², and so on, including the bonus after full acceptance.

expected tokens = 1 + α + α² + … + α^k

Measure time in units of one ordinary target decoding step. Each draft step costs c, and verification costs v. Then the round costs v + kc, and:

speedup = expected tokens / (v + kc)

Acceptance α80% Draft length k4 Draft step cost c0.10 Verification cost v1.00

3.36 expected tokens÷1.40 target-step units=2.40× speedup

Scroll sideways to see draft lengths 1–12.

Simplified latency model, not measured performance. The curve holds α, c and v constant as draft length changes. High acceptance helps only when it repays the time spent drafting and verifying.

With α = 0.8 and four proposals, this model yields 3.36 expected tokens per round. If a draft step costs 0.10 target steps and verification costs 1.00, the modeled speedup is 3.36 / 1.40 ≈ 2.40×.The original analysis, §3, uses v = 1. This widget exposes v to show how a more expensive verification pass changes the result. Acceptance here is distribution overlap, not benchmark accuracy.

The model ignores setup, communication, cache management, and the end of a finite response. In a real workload, acceptance varies with context and verification cost can grow with draft length. Increasing the serving batch size can also reduce the spare compute that made verification cheap.

What stays fixed, and what can change

The target remains the authority over output. The draft can be less capable, provided the acceptance and correction rules are implemented correctly. A weak draft usually wastes time; it does not deliberately lower the target’s sampling quality.

Property	In the basic draft–target algorithm
Greedy output	Same token choices with consistent tie-breaking and equivalent numerics.
Sampled output	Same distribution; not necessarily the same random realization.
Memory	The draft model and its state add memory; rejected cache entries must be rolled back or excluded.
Vocabulary	Both distributions are defined over the same token vocabulary.
Stopping	Honor end-of-sequence and length limits on committed output; do not emit tokens beyond them.
Latency	Depends on acceptance, drafting cost, verification cost and the serving workload.

This tutorial covers the original linear draft–target method. Other systems propose token trees, use extra prediction heads, or obtain drafts without a separate small model. Their schedules and verification rules need their own analysis. The useful question remains concrete: how much valid output does each expensive target pass buy?