Skip to content

35 min read by CloudMash

Your GPU is bored: speculative decoding from first principles to EAGLE-4

A small model guesses the next few words, the big model checks them all at once, and you get the same answer in a fraction of the time. Here is how, from the core idea to EAGLE-4.

A trick that lets a language model write several words at a time without changing a single word of its answer, explained slowly with one sentence about a cat, then followed all the way to EAGLE-4.

One word at a time 4 big steps step 1 big model step 2 big model step 3 big model step 4 big model Guess a few, check once 1 big step small model quick guesses ? ? ? ? big model checks all 4 guesses in 1 step 3 guesses kept, 1 replaced Same answer, fewer big steps big model small model guess kept replaced by big model

One word at a time

When a language model answers you, it writes one word at a time. (Strictly, it writes tokens, small chunks of text that are often whole words. In this post you can read "token" as "word".)

To write each word, the model runs its whole network once, start to finish. For a model with tens of billions of internal numbers (its parameters), that is a lot of work for a single word. Then it does it all again for the next word, and the next.

And most next words are boring. After "The cat sat on the", you already know the next word is probably "mat". After "Thank you very", it is "much". The model still runs its entire network to find that out, one word per run.

So here is the question this post is about: if the next few words are that obvious, why not let something cheap guess several of them, and have the big model only check the guesses?

That is speculative decoding. The surprising part, which we will get to, is that checking four guesses takes the big model about as long as writing one word. When the guesses are good, you get several words for the price of one, and the answer comes out exactly the same.

Guess, then check

This is the heart of the idea, so we will take one round slowly, with one example that the rest of the post keeps coming back to.

The two models and the sentence

Two models take part.

The target is the big model, the one whose answers we actually want. In this post it is Qwen3.8-27B, an open model with 27 billion parameters released in August 2026.

The drafter is a much smaller and faster model. Its only job is to guess what the target would write next. It does not have to be right; it only has to be right often.

The text so far is:

The cat sat on the

The drafter guesses four words, one after another: "mat", then "and", then "fell", then "asleep". These are sensible guesses. Cats sit on mats, "mat and" is a natural way to go on, and a cat on a mat might well fall asleep.

But the target, it turns out, would have written "The cat sat on the mat and purred". So the drafter's first two guesses are right and its third is wrong.

One round, step by step

Step 1, guess. The drafter writes its four guesses, one at a time: "mat", "and", "fell", "asleep". It is small, so each guess is quick.

Step 2, check, all at once. The target reads the original text with all four guesses attached, "The cat sat on the mat and fell asleep", in a single run. Here is the key fact: in that one run, the target works out what it would have written at every position along the way, not just at the end. It does exactly the same thing whenever it reads your prompt. (Why one run checks every position is in the deep dive at the end.) So after one run, the target knows that after "the" it would write "mat", after "mat" it would write "and", and after "and" it would write "purred".

Step 3, compare from left to right. "mat" matches what the target wanted: keep it. "and" matches: keep it. "fell" does not: the target wanted "purred". So "fell" is thrown away, and so is "asleep", because it was built on top of a wrong word. In their place goes the target's own word, "purred". That correction costs nothing extra: the target already worked it out during the same run.

The round ends with three new words, "mat and purred", from a single run of the big model. The next round starts from "The cat sat on the mat and purred", and the drafter guesses again.

Here is that same round as a picture. The drafter's four guesses (the candidates) sit on top, and the target's check is below. Green means kept, red marks the first wrong guess, gold is the target's correction, and the dimmed word is the one thrown away.

ONE ROUND 4 draft passes + 1 target pass CONTEXT The cat sat on the DRAFTER small . 4 fast passes CANDIDATES 4 draft tokens TARGET one pass checks all 4 ACCEPT prefix + correction THIS ROUND k = 4 the drafter proposes, one pass each mat and fell asleep ok ok no out the target checks all 4 in one pass mat and purred 3 tokens, 1 target pass 2 accepted + 1 correction, taken free "fell" rejected: the target said "purred" "asleep" came after a mistake: thrown away all 4 right would add a 5th, bonus token ONE ROUND 4 draft passes + 1 target pass CONTEXT The cat sat on the DRAFTER small . 4 fast passes CANDIDATES 4 draft tokens TARGET one pass checks all 4 ACCEPT prefix + correction THIS ROUND k = 4 the drafter proposes, one pass each mat and fell asleep ok ok no out the target checks all 4 in one pass mat and purred 3 tokens, 1 target pass 2 accepted + 1 correction, taken free "fell" rejected: the target said "purred" "asleep" came after a mistake: thrown away all 4 right would add a 5th, bonus token
One round of the example: four quick guesses, one run of the big model, two guesses kept, the third replaced by the big model's own word, and the fourth thrown away.

The same round as a table, one row per position:

positiontarget saysdrafter saidresult
after "the"matmatkeep
after "mat"andandkeep
after "and"purredfellwrong: use "purred"
after "fell"(not needed)asleepthrown away: it came after a mistake

What one round saves

Give the two models round, made-up costs: say one run of the target takes 50 ms and one guess from the drafter takes 5 ms.

Without guessing, the target writes "mat" (50 ms), then "and" (50 ms), then "purred" (50 ms): three words in 150 ms.

With guessing, the drafter makes four guesses (4 x 5 ms = 20 ms) and the target checks them in one run (50 ms): the same three words in 70 ms.

work donetimenew wordstime per word
no guessing3 target runs150 ms350 ms
guessing, this round4 guesses + 1 target run70 ms323.3 ms (2.14x faster)
guessing, all 4 guesses right4 guesses + 1 target run70 ms514 ms (3.57x faster)
guessing, first guess wrong4 guesses + 1 target run70 ms170 ms

Three things to notice

  • You always get at least one new word per target run. Even if the very first guess is wrong, the target's correction counts. The worst case wastes only the drafter's 20 ms.
  • If every guess is right, you get a bonus word. The target's single run also tells it what comes after the last guess. Four right guesses plus that bonus is five words for one run, which is the third row of the table.
  • How often the guesses are right decides everything. This is called the acceptance rate. If the drafter is usually right, most rounds keep three or four guesses and the speedup is large. If it is usually wrong, most rounds keep only the correction and the guessing time is wasted. With guesses that are right 80% of the time and four guesses per round, a round produces about 3.4 words on average. (The formula is in the deep dive.)

And one thing that never changes: the words that come out are exactly the words the target would have written on its own. The drafter only proposes; the target decides. A later section shows why this holds even when the target picks its words randomly.

Try it

Now that green and red mean something, play with the two numbers that matter. Acceptance rate is how often a guess is right. Draft length is how many words the drafter guesses per round. The two cost boxes are the time per guess and per target run (5 ms and 50 ms, as above). Leave batch pressure at 1 for now; it comes back in the section on production. Press "run a round" to watch one round play out with random guesses.

// interactive toy model

Speculative decoding playground

Set how often the drafter guesses right and how far ahead it guesses, then run a round and watch the target model check every guess in one pass.

Chance that each drafted token matches what the target would have picked.
Multiplies the target pass cost T, to mimic a GPU that is busy with many requests and no longer bandwidth-bound.
D = 5 ms and T = 50 ms are illustrative costs, not measurements.
tokens per target pass
3.36
speedup
2.40x
speculative ms per token
20.8 ms
vanilla ms per token
50.0 ms

step = 4 x 5 + 50 x 1.00 = 70.0 ms

1x 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 2.47x
Speedup for each draft length k at a = 0.80. Best: k = 6, 2.47x. Outlined bar: your k.

context The cat sat on the

matandfellasleep

Press "run a round": the drafter has proposed 4 tokens, the target will check them all in one pass.

Total: 0 tokens over 0 rounds.

A constant a is a simplification: real engines (EAGLE-2 and later) use the drafter's own confidence for each token, which is why draft trees beat a single chain.

Two things show up quickly. The best number of guesses grows as the guesses get better: guessing far ahead only pays when you guess well. And when guesses are often wrong, the speedup collapses, because you pay for every guess and keep almost none. Real systems are smarter than a fixed hit rate: from EAGLE-2 on, they use how confident the drafter is about each word, which is where this post is heading.

In two sentences: a small model guesses the next few words, and the big model checks all of them in a single run, keeps the ones it agrees with, and puts its own word where the first wrong guess was. You get at least one word per run of the big model and often several, and the text is exactly what the big model would have written alone.

Everything else in this post, including the whole EAGLE family, is about Step 1: making the guesses better, so more of them are kept per round.

Why checking is almost free

Why does checking four guesses cost the target about the same as writing one word? Because of what actually slows a big model down.

To produce anything at all, the model has to read every one of its parameters from the GPU's memory: for Qwen3.8-27B, 27 billion numbers, about 54 GB. Reading them takes far longer than the arithmetic done with them. While a model writes one word at a time, the GPU spends most of each run waiting for numbers to arrive from memory.

Once the numbers are loaded, using them for four words instead of one adds a little arithmetic and almost no time. The extra words ride along in the same transformer run, using compute that was sitting idle anyway. On an H100, reading those 54 GB alone takes about 16 ms per run, however many words are being checked. (The full bandwidth math is in the deep dive.)

The diagram shows the two cases side by side: one word per run and seven words per run, with the same 54 GB of weights flowing through the same pipe from memory to the compute units.

QWEN3.8-27B . BF16 . ONE H100 SXM one decoding step reads every weight once 1 TOKEN PER STEP the compute mostly waits HBM3 54 GB weights of 80 GB on the card 54 GB a step about 16 ms COMPUTE x1 7 TOKENS PER STEP x7 the work, same bytes HBM3 54 GB weights of 80 GB on the card 54 GB a step about the same COMPUTE x7 BACK OF THE ENVELOPE 27B params x 2 bytes (bf16) = 54 GB a step H100 SXM: 54 GB / 3.35 TB/s = about 16 ms = at most about 62 tokens/s for one stream a floor from bandwidth alone, before KV reads QWEN3.8-27B . BF16 . ONE H100 SXM one decoding step reads every weight once 1 TOKEN PER STEP the compute mostly waits HBM3 54 GB weights 54 GB a step about 16 ms COMPUTE x1 7 TOKENS PER STEP x7 the work, same bytes HBM3 54 GB weights 54 GB a step about the same COMPUTE x7 BACK OF THE ENVELOPE 27B params x 2 bytes (bf16) = 54 GB a step H100 SXM: 54 GB / 3.35 TB/s = about 16 ms = at most about 62 tokens/s for one stream a floor from bandwidth alone, before KV reads
The same 54 GB of weights is read on every run. With one word the compute units mostly wait; with seven, they finally have something to do, and the run takes about as long.

That is the free lunch. The guesses are cheap, and checking them is almost free, so the only question left is how to guess well.

Why the output is exactly the same

When the target always picks its single most likely word, this is easy to see: keep a guess only if it equals the target's choice, and the text is word for word what the target would have written alone.

Most chat models pick words with some randomness, though. Then the rule is a little cleverer. A guess is kept with a probability that depends on how much the target likes that word compared with how much the drafter liked it: if the target likes it at least as much, it is always kept; if less, it is kept only some of the time. When a guess is rejected, the replacement is drawn from the words the drafter gave too little weight to.

A tiny example shows that the bookkeeping works out exactly. Imagine a vocabulary of three words, A, B and C. The target wants them 50%, 30% and 20% of the time. The drafter, slightly off, guesses them 20%, 50% and 30% of the time.

wordtarget wantsdrafter guesseschance a guess is keptgets the rejected share
A0.50.21all of it
B0.30.50.6none
C0.20.30.667none

A comes out when the drafter guesses A (0.2, always kept) plus every rejection (0.5 x 0.4 + 0.3 x 0.333 = 0.3), so 0.5. B comes out when the drafter guesses B and it is kept: 0.5 x 0.6 = 0.3. C: 0.3 x 0.667 = 0.2. That is exactly what the target wanted. The drafter's mistakes cost speed, never correctness. (The exact rule is in the deep dive.)

The bars show the same example: what the target wants, what the drafter guesses, and how kept guesses plus resampled rejections add up to the target's numbers.

target p what the big model wants A 0.5 B 0.3 C 0.2 draft q what the drafter proposes A 0.2 B 0.5 C 0.3 accept: min(1, p / q) chance a drafted token is kept A 1 B 0.6 C 0.667 residual: max(0, p - q) normalised; used on the 30% of rejections A 1 B 0 C 0 output = p kept (green) + resampled (gold) A 0.5 B 0.3 C 0.2
Target p, drafter q and the leftover p - q for the three-word example. Kept guesses plus resampled rejections add up to exactly p.

In practice, vLLM calls its sampler "theoretically lossless up to the precision limits of hardware numerics" and warns that floating point and batching effects can still make outputs differ bit for bit (vLLM docs). The math is exact; the hardware is not quite.

The drafter problem

The trick was found in 2022 by two teams independently: Yaniv Leviathan, Matan Kalman and Yossi Matias at Google (arXiv 2211.17192), and Charlie Chen and colleagues at DeepMind (arXiv 2302.01318). Google reported a 2x to 3x speedup on T5-XXL with identical outputs; DeepMind 2x to 2.5x on Chinchilla 70B. In both, the drafter was simply a smaller language model.

That recipe needs a second, smaller model with the same tokenizer, trained separately, that happens to agree with the big one. That is clumsy. A 1B model drafting for a 70B model has never seen the big model's internal state; it guesses from the visible tokens alone. It takes its own memory. It is not free: k draft passes per step. And for many model families there is no small sibling at all.

Making the drafter bigger is the obvious fix, and it mostly does not work: a bigger drafter guesses better but costs more per guess. The EAGLE team made exactly that point in a June 2026 thread, and the EAGLE-4 paper puts numbers on it for our model. The 2026 block drafters for Qwen3.8-27B, DFlash 2 and DSpark, stack 5 layers (about 1.66B and 1.65B layer parameters per drafted position), while EAGLE-4 uses 1 layer (0.44B), spending about 3.8x fewer FLOPs per drafted token (paper, Table 1 and Section 1).

Think back to our sentence. A separate small model guessing "mat", "and", "fell", "asleep" only ever sees the words "The cat sat on the". It has no idea what the target was actually thinking when it read them, which is why its third guess went wrong. So the real question is different: what if the drafter could read the big model's mind instead of its mouth?

EAGLE-1: draft from the model's thoughts

EAGLE (arXiv 2401.15077, ICML 2024) starts from two observations.

First, predicting the target's next feature (the hidden state just before the LM head, from its second-to-top layer) is much easier than predicting its next token. Features change smoothly from step to step; tokens jump around.

Second, features carry a built-in uncertainty: the next feature depends on which token was actually sampled, and sampling is random. EAGLE's fix is to feed the drafter the token sequence shifted one step ahead alongside the features, so it always knows which token was picked.

The drafter is a single lightweight decoder layer, the EAGLE head. It takes the current feature plus the embedding of the next token and predicts the next feature; the target's own frozen LM head turns that predicted feature into a token distribution. The target's weights are never touched; only the small head is trained, on the target's own features.

Back to our sentence. When the target finishes reading "The cat sat on the", its last-layer feature already leans strongly toward "mat". The EAGLE head reads that feature plus the token "mat" and predicts the feature for the next position, where "and" is the likely word, without running the 27-billion-parameter stack again. That is why its guesses are good where a small sibling model's guesses are mediocre.

The diagram shows exactly that: the target's tall stack of layers, a tap on its second-to-top layer feeding the small EAGLE head beside it, the token "mat" joining from below, and the target's own LM head on top turning the head's prediction into the next word, "and".

target picks mat draft token and LM head shared by both, frozen 64 . attention 63 . DeltaNet 62 . DeltaNet 61 . DeltaNet 3 . DeltaNet 2 . DeltaNet 1 . DeltaNet 57 more layers Qwen3.8-27B 64 layers, frozen context The cat sat on the tap: f(t) 2nd-to-top feature, after 'the' EAGLE head 1 layer, trainable f(t) + emb(t+1) f(t+1) predicted feature loop: next step mat token t+1, embedded shifted one step: the target's own pick
EAGLE-1: a one-layer head taps the target's second-to-top layer, adds the embedding of the token that was just picked, predicts the next feature, and reuses the target's frozen LM head to turn it into a token.

The paper reports a 2.7x to 3.5x latency speedup on LLaMA2-Chat 70B and doubled throughput. The repository's 13B demo puts EAGLE-1 at 3x over plain decoding, 2x over Lookahead and 1.6x over Medusa.

EAGLE-2: dynamic draft trees

A chain of guesses has a weakness: one wrong token kills everything after it. Look at our example. After "and", a reasonable drafter is torn between "fell" and "purred". A chain must bet on one, and our chain bet on "fell". A tree can send both.

EAGLE-1 already used trees, but with one fixed shape for every context. EAGLE-2 (arXiv 2406.16858, EMNLP 2024) noticed that the drafter's confidence (the probability it assigns to its own guess) is well calibrated: it predicts how likely the target is to accept. So the tree can be shaped per context, in two phases:

  • Expand: grow the tree by extending the most promising nodes, where a node's score is the product of the draft probabilities along its path.
  • Rerank: pick the top nodes overall and send those for verification. Any prefix of that ranked list is itself a valid tree.

The target still checks everything in one run. It reads the whole tree at once, and a mask makes sure each guess only looks at the words on its own branch, so "fell" and "purred" are checked side by side, as if each were the only way to continue. (How the tree attention mask works is in the deep dive.)

The diagram shows our sentence as a tree: "mat" and "and" as a shared trunk, then a split into "fell" (with "asleep" after it) and "purred". Beside the tree is the mask, a grid showing which guesses each guess is allowed to look at.

EAGLE-2 DRAFT TREE after one target pass The cat sat on the mat 1 and 2 fell 3 asleep 4 purred 5 rejected fell was wrong accepted target's pick chain: only "fell" sent, 2 accepted + correction tree: "purred" sent too, 3 accepted + bonus drafter confidence, illustrative values TREE ATTENTION MASK filled = the row may attend to the column P 1 2 3 4 5 1 mat 2 and 3 fell 4 asleep 5 purred P = the prompt: every node sees it 5 (purred) cannot see 3 (fell) or 4 (asleep) one target pass checks all 5 nodes EAGLE-2 DRAFT TREE after one target pass The cat sat on the mat 1 and 2 fell 3 asleep 4 purred 5 rejected fell was wrong accepted target's pick chain: only "fell" sent, 2 accepted + correction tree: "purred" sent too, 3 accepted + bonus drafter confidence, illustrative values TREE ATTENTION MASK filled = the row may attend to the column P 1 2 3 4 5 1 mat 2 and 3 fell 4 asleep 5 purred P = the prompt: every node sees it 5 (purred) cannot see 3 (fell) or 4 (asleep) one target pass checks all 5 nodes
The running example as a dynamic tree: the trunk is shared, the tree branches where the drafter is unsure, and the target accepts the 'purred' branch instead of rejecting a single chain. Alongside it: the tree attention mask that verifies every branch in one pass.

With a tree, the third guess is no longer a rejection: the target picks "purred", that branch is accepted, and the round commits "mat and purred" as three accepted drafts plus a bonus token from the same pass. EAGLE-2 reports 3.05x to 4.26x across three model series and six tasks, 20% to 40% faster than EAGLE-1, with the same drafter and no retraining: the gain is entirely in how the tree is built.

EAGLE-3 and EAGLE 3.1

EAGLE-1 and 2 trained the head to reproduce the target's next feature. That objective was also a ceiling: forcing the head to copy a feature limits how much it improves with more data. EAGLE-3 (arXiv 2503.01840, NeurIPS 2025) made three changes.

  • Predict tokens directly. Drop the feature-regression loss and train the head on the thing that matters, the next token.
  • Multi-layer feature fusion. Instead of one tap at the second-to-top layer, concatenate low-, mid- and high-level features from several target layers, then project them down. Early layers know about syntax and copying, late layers about meaning; the head gets all of it.
  • Training-time test. At inference, the head's later draft steps consume the head's own previous outputs, not fresh target features. EAGLE-1 and 2 only ever trained on real target features, so they were tested on inputs they never saw in training. EAGLE-3 simulates multi-step drafting during training, so training matches use.

That last change is the one that lets acceptance keep growing as you add training data.

For our sentence, the difference is in the third guess. EAGLE-1 had one view of the target's thinking after "The cat sat on the mat and"; EAGLE-3 sees several layers of it at once and was trained on chains of its own guesses, so it is better placed to rank "purred" above "fell" in the first place.

The diagram shows the three taps at different heights of the target's stack merging into one input for the head, and a dotted loop where the head is trained on its own previous outputs.

high layer mid layer low layer Target . Qwen3.8-27B 64 layers, frozen h m l concat then project g fused vector \n token emb EAGLE-3 head 1 layer predicts tokens if training-time test: train on your own outputs EAGLE-1/2 1 top tap . regress the feature EAGLE-3 3 taps fused . predict tokens
EAGLE-3: three taps (low, mid, high) fuse into one vector for the head, which predicts tokens directly; the dotted loop is training-time test, where the head trains on its own previous outputs.

The abstract reports up to 6.5x speedup, about 1.4x over EAGLE-2, and 1.38x throughput at batch size 64 in SGLang. SGLang's docs show the progression on one H100 with Llama-3.1-8B-Instruct on MT-Bench: 158.34 tokens/s without speculation, 244.10 with EAGLE-2, 373.25 with EAGLE-3 (SGLang docs).

EAGLE 3.1 (May 2026, the EAGLE team with vLLM and TorchSpec, vLLM blog) fixed something subtle the team calls attention drift: as the draft gets deeper, the head attends less to the important early tokens and more to its own guesses, partly because the higher-layer features dominate the fused input and the hidden state grows in magnitude step after step. Two small changes, a normalization of each target feature before fusion and post-normalized hidden states fed into the next step, keep the feedback loop stable. The post reports long-context acceptance length up to 2x that of EAGLE-3, and on Kimi-K2.6 NVFP4 with vLLM on GB200 (TP4), 2.03x per-user throughput at concurrency 1, 1.71x at 4 and 1.66x at 16. The work, described in the paper "Attention drift" (arXiv 2605.09992), was accepted to NeurIPS 2026. The team announced it on X.

EAGLE-4: one drafter for every batch size and every chip

Every method so far has a hidden assumption: that there is one right amount of drafting. There is not. At batch size 1 the GPU is idle, so you should draft deep and wide. At 512 concurrent requests the target pass is compute-bound, every extra verified token costs real time, and the right answer is a token or two. Drafters with a fixed block size cannot adapt, and the EAGLE-4 paper shows what happens: at 512 concurrent requests, the 2026 block drafters fall below plain decoding (0.78x and 0.67x).

EAGLE-4 (Eagle AI, released October 20, 2026; paper, repo) attacks that with three ideas.

EAGLE-Harness: measure the hardware, then decide

At startup, the harness profiles two costs on the machine it is running on: the time of a draft step, and the time to verify a given number of tokens, indexed by batch size and context length. It caches those tables. During decoding, every decision is a single comparison: extend the tree one more level, or verify one more token, only if the expected gain in accepted tokens is worth the measured time:

deltaA / deltaC >= tau

The threshold tau rises with depth, because deep estimates are less reliable (Section 3.1, Algorithm 1). Under light load the tree may branch up to 8 ways per node; under heavy load at most 4. The paper's configurations show the range: depth 12, width 4 and 16 verified tokens at concurrency 1, versus a 15-deep chain of width 1 at concurrency 512. Same weights: wide, branching trees for one user, a single narrow chain when the server is full. No retraining, only table lookups and scores the drafter already computes.

For our sentence, that means one person typing "The cat sat on the" might get a wide tree that tries both "fell" and "purred" after "and", while on a server handling 512 conversations the same drafter sends one guess per position and spends nothing on branches that will probably be thrown away.

The diagram shows the loop: cost tables measured once, then for every step the single comparison that decides whether to guess further, how many guesses to send for checking, and the check itself. The inset compares the tree built for one request with the one built for 512.

profile once, at startup draft-step and verify cost tables, keyed by batch size and context length cost lookups draft step one more level deltaA / deltaC >= tau ? yes deepen / extend no stop drafting choose verify width keep nodes worth their cost target verify one pass, every chosen token commit accepted prefix + correction next round same weights, two shapes batch 1 deep and wide, up to 8 branches per node batch 512 short chain, at most 4 branches per node verify profile once, at startup draft-step and verify cost tables, keyed by batch size and context length cost lookups draft step one more level deltaA / deltaC >= tau ? yes deepen / extend no stop drafting choose verify width keep nodes worth their cost target verify one pass, every chosen token commit accepted prefix + correction next round same weights, two shapes batch 1 deep and wide, up to 8 branches per node batch 512 short chain, at most 4 branches per node verify
The EAGLE-Harness loop: cost tables measured once per machine, then one comparison per decision. Inset: the tree it builds for one request versus 512.

The Efficient LM Head

Once the drafter is one layer, the slowest part of it is the vocabulary. Scoring all 248,320 entries of Qwen3.8-27B's vocabulary was 65.4% of a 574 microsecond draft step (paper, Figure 2). EAGLE-4 adds a low-rank head (rank 128 to 512) that only shortlists K candidates (128, 256 or 512) from the full vocabulary, then rescores exactly those K rows with the real LM head. The work drops from O(V d) to O(d r + V r + K d), and the draft step from 574 microseconds to 326 (rank 512) and 255 (rank 128), cuts of 43.2% and 55.6%.

The trick is training the two low-rank factors for retrieval. A plain SVD initialization at rank 128 lost 0.48 tokens of acceptance length per verification; the trained head keeps the full head's top choice in its shortlist 98.2% of the time (91.5% for SVD) and shows no acceptance loss. Draft time per verification cycle halves, from 9.1 ms to 4.8 ms at batch size 1. The drafter also uses sliding-window attention for long contexts, an append-only cache, and a feed-forward layer that mirrors the target, dense or MoE.

The results

On Qwen3.8-27B with four B300 GPUs (TP4, SPEED-Bench, paper Table 2):

method1 request, T = 01 request, T = 1512 requests, T = 0
plain decoding240.5 tok/s234.921,203
built-in MTP head534.5 (2.22x)369.1 (1.57x)21,518 (1.01x)
DSpark618.6 (2.57x)519.4 (2.21x)14,229 (0.67x)
DFlash 2672.6 (2.80x)560.3 (2.39x)16,641 (0.78x)
EAGLE-4863.8 (3.59x)608.6 (2.59x)28,340 (1.34x)

On HumanEval at one request it reaches 929 tok/s (Figure 3). Vendors that measured it on their own stacks report similar shapes: AMD sees about 4.7 tokens accepted per check against about 2.9 for the built-in MTP head (AMD), and LightSeek reports 985 tok/s greedy on TokenSpeed with the same four B300s (LightSeek). The team's own launch post quotes a higher single-request number with TokenSpeed; the paper's figures are the ones above.

Then the part that makes this fun. The same algorithm runs well away from the data center:

  • One AMD MI350X (paper Table 4): 75.27 to 262.89 tok/s at one request (3.49x), still 1.70x at 32. AMD shipped day-0 support on MI355X: 82 to 289 tok/s in SGLang and 309 on its ATOM engine for one request.
  • A 4-vCPU cloud CPU (AWS c5.xlarge, 8 GB, llama.cpp, 4-bit Gemma-4-E2B-it, Table 5): 2.28x to 2.92x, 2.71x on average, where EAGLE-3 managed 1.25x to 1.65x on the same box. The whole draft head is 134 MB; total extra memory 0.27 GB.
  • A phone (Motorola Razr 70 Ultra, Snapdragon 8 Elite, unmodified and not rooted, llama.cpp on 6 threads, Table 6): HumanEval from 22.8 to 82.6 tok/s (3.62x), long-form writing 23.3 to 74.7 (3.21x). Energy per generated token fell from 356 mJ to 166 mJ, measured at the USB input. A generation that took 43 seconds takes 14.

The algorithm that feeds four B300s also halves the battery cost of a token on a phone.

If you were waiting for a speculative-decoding chip: as of November 2026 no primary source announces one from Eagle AI. The more interesting news is the opposite. The drafter now treats the chip as a parameter: NVIDIA sm_80 and newer, AMD CDNA2 to CDNA4, x86 CPUs, an Arm phone.

Go deeperlicense, EAGLE-5, the wider EAGLE footprint

EAGLE-4 is source-available, not OSI open source: commercial use above a 20 million US dollar aggregate-revenue threshold, or any competing offering, needs a separate license (README). The earlier EAGLE-1, 2 and 3 code (SafeAILab/EAGLE) is Apache 2.0. GMI Cloud has announced a closed, hosted successor called EAGLE-5 (GMI Cloud).

EAGLE ideas also run well outside the EAGLE repos. NVIDIA's record of over 1,000 tokens per second per user on Llama 4 Maverick on one DGX B200 used a drafter trained with EAGLE-3 techniques; speculation alone gave about 2.06x at draft length 3 (NVIDIA). On Jetson AGX Thor, EAGLE-3 is the biggest piece of a 7x cumulative gain on Llama 3.3 70B, and 2.5x on its own at W4A16 and one request (NVIDIA).

The wider family

EAGLE is one branch of a busy tree. Each of these solves a different piece of the problem.

Medusa (arXiv 2401.10774) skips the separate drafter: several extra LM heads sit on the target's last hidden state, each predicting the token k steps ahead, and their candidates are verified with tree attention. Medusa-1 trains only the heads (over 2.2x); Medusa-2 fine-tunes the backbone too (2.3x to 3.6x), so it is no longer strictly lossless. Hydra (arXiv 2402.05109) makes those heads depend on each other's guesses, for up to 1.31x over Medusa.

SpecInfer (arXiv 2305.09781) uses small models to build a token tree and verifies it in one tree-parallel pass: 1.5x to 2.8x for distributed serving, 2.6x to 3.5x with offloading.

Lookahead decoding (arXiv 2402.02057) needs no drafter at all: Jacobi iteration guesses several future positions at once, an n-gram pool collects what it finds, and the same pass verifies them. Up to 1.8x on MT-Bench and up to 4x on multi-GPU code completion.

Prompt lookup (repo) is the cheapest drafter possible: find the last few tokens earlier in the prompt and propose what followed them. 2.4x on average for summarization and document QA with Mistral-7B-Instruct on an A100. It shines when the output repeats the input: in a code edit or a summary, the next words often already exist in the prompt. That is also why wild copy-workload anecdotes circulate online: when the output is a copy of the input, nearly every guess is accepted.

Self-speculative decoding drafts with the model itself. Draft & Verify (arXiv 2309.08168) skips layers chosen by Bayesian optimization, with no training, for up to 1.99x. LayerSkip (arXiv 2404.16710) trains the model with an early-exit loss so its early layers can draft: up to 2.16x on summarization, 1.82x on code.

ReDrafter from Apple (arXiv 2403.09919) uses a small RNN head with beam search: up to 2.8x for Vicuna on an H100, 2.3x with MLX on Apple Silicon.

Multi-token prediction heads turn drafting into a training feature. DeepSeek-V3 (arXiv 2412.19437) trains one extra MTP module and keeps it at inference: the second token is accepted 85% to 90% of the time, for 1.8x tokens per second. The EAGLE-4 paper points out that mainstream MTP heads follow EAGLE's design of concatenating the target's hidden state with the token embedding; Qwen3.8-27B's own head is the 534.5 tok/s row in the table above.

methoddrafterextra traininglosslessheadline number (setting)best for
Leviathan et al.small modelnone, needs a siblingyes2x to 3x (T5-XXL)any model with a small sibling
Chen et al.small modelnone, needs a siblingyes2x to 2.5x (Chinchilla 70B)distributed serving
SpecInfersmall models, treesmall draftersyes1.5x to 2.8x distributeddistributed and offloaded serving
Medusa-1 / Medusa-2extra headsheads (M2 also backbone)M1 yes, M2 noover 2.2x / 2.3x to 3.6xone model, batch 1
Hydradependent headsheadsyes1.31x over MedusaMedusa upgrade
Lookaheadnone (Jacobi)noneyesup to 1.8x (MT-Bench)no drafter, spare FLOPs
Prompt lookupprompt n-gramsnoneyes2.4x avg (Mistral-7B, A100)RAG, summaries, code edits
Draft & Verifyskipped layersnoneyesup to 1.99x (LLaMA-2)zero extra memory
LayerSkipearly exitretrain targetyesup to 2.16x (CNN/DM)one model, tight memory
ReDrafterRNN headsmall headyesup to 2.8x (Vicuna, H100)NVIDIA and Apple Silicon
DeepSeek-V3 MTPbuilt-in headin pretrainingyes1.8x TPSmodels trained with MTP
EAGLE-1feature headsmall headyes2.7x to 3.5x (LLaMA2-Chat 70B)chat, code, low batch
EAGLE-2same head, dynamic treenone beyond EAGLE-1yes3.05x to 4.26xcontext-dependent text
EAGLE-3fused-feature headsmall head, more datayesup to 6.5xchat and reasoning
DFlash 25-layer block diffusionyesyes2.80x (Qwen3.8-27B, 4x B300)batch 1, long blocks
DSpark5-layer semi-autoregressiveyesyes2.57x (Qwen3.8-27B, 4x B300)batch 1, production schedulers
EAGLE-41-layer head + harnesssmall headyes3.59x at 1 request, 1.34x at 512every batch size and chip

The last three rows are from the EAGLE-4 paper's Table 2 on Qwen3.8-27B; the rest are each paper's own headline.

The block drafters of 2026

The strongest rivals to EAGLE this year do not draft one token at a time. They draft a whole block in one pass.

DFlash (arXiv 2602.06036, z-lab/dflash, MIT) makes the drafter a small block-diffusion model that produces a whole block of tokens at once, conditioned on the target's features; its abstract reports over 6x lossless acceleration and up to 2.5x the speedup of EAGLE-3. DFlash 2 (Inco AI, August 2026) adds a path selector that walks one coherent path through the top 16 candidates per position, plus a small dynamic convolution, for about 1.3% extra latency; Inco reports 2.7x to 3.4x on Qwen3.8-27B in SGLang at batch size 1. Its author announced a DFlash 2 drafter for Qwen3.8-Flash-Next in October.

DSpark (arXiv 2607.05147, DeepSeek-AI and Peking University, code at deepseek-ai/DeepSpec) pairs a parallel DFlash-style backbone with a light sequential "Markov head" for dependencies inside the block, and schedules verification per request from estimated survival probabilities and the engine's current load. In DeepSeek-V4 production serving it makes per-user generation 60% to 85% faster than an MTP-1 baseline. People are running it in surprising places: Zhihao Jia reports Qwen3.8-27B at over 200 tokens per second per user on a single Apple M5 Max with megakernels plus DSpark, his own measurement.

The upside of block drafting is that drafting time does not grow with block length. The downside is the EAGLE-4 paper's argument (Section 1.1): a block drafter has paid for the whole block before it learns which tokens were useless, while an autoregressive drafter can stop before paying for the next step. At large batch the best block shrinks to one or two tokens, where parallel drafting has no advantage left. That is the 0.67x and 0.78x in the table.

Go deeperHASS, GliDe, FR-Spec, PARD, SuffixDecoding, trainers and benchmarks
  • HASS (arXiv 2408.15766, ICLR 2025): trains EAGLE-style heads with harmonized objective distillation and context alignment; 2.81x to 4.05x, 8% to 20% over EAGLE-2.
  • GliDe with a CaPE (arXiv 2402.02082): the drafter reuses the target's KV cache, and confidence adds extra candidates; up to 2.17x, 2.61x with CaPE.
  • FR-Spec (arXiv 2502.14856): restricts the drafter's LM head to a frequency-ranked vocabulary subset, cutting LM-head compute 75%, 1.12x over EAGLE-2. It is the idea EAGLE-4's Efficient LM Head replaces with an exact rescoring.
  • PARD (AMD, arXiv 2504.18583): one target-independent parallel draft model per model family; up to 3.67x on Llama-3.1-8B in vLLM.
  • SuffixDecoding (Snowflake, arXiv 2411.04975): model-free suffix trees over prompts and earlier outputs; up to 5.3x on SWE-Bench and text-to-SQL agent workloads.
  • Trainers: SpecForge (SGLang's MIT trainer for EAGLE3, EAGLE 3.1, DFlash, DFlash 2 and DSpark drafters) and TorchSpec (LightSeek, MIT, disaggregated training; LightSeek says it trained the EAGLE-4 drafters).
  • Benchmarks: Spec-Bench (2024; EAGLE-3 at 3.02x with 5.71 mean accepted tokens on Vicuna-13B on an A100) and SPEED-Bench (NVIDIA, arXiv 2604.09557), which the 2026 papers use.

In production

All the major engines speak speculation now.

vLLM takes a --speculative-config JSON whose method includes draft_model, ngram, eagle3, mtp, suffix, dflash and more (docs). The zero-download way to try it is n-gram drafting from the prompt:

vllm serve <target-model> \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

SGLang uses --speculative-algorithm with EAGLE, EAGLE3, DFLASH, STANDALONE, NGRAM and UNO, and three knobs that map straight onto this post: --speculative-num-steps (depth), --speculative-eagle-topk (branching) and --speculative-num-draft-tokens (how many tokens to verify) (docs). The docs' EAGLE example:

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

EAGLE-4 plugs into SGLang, vLLM and TokenSpeed through the eagle-harness package; for our model the README's short form is:

sglang serve --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4

TensorRT-LLM (PyTorch backend) has Eagle3DecodingConfig, MTPDecodingConfig, DraftTarget, NGram (deprecated), PARD, DFlash and SA (docs). The MTP path for a model with native MTP modules is three lines:

from tensorrt_llm.llmapi import MTPDecodingConfig

speculative_config = MTPDecodingConfig(max_draft_len=3)

llm = LLM("/path/to/deepseek_model", speculative_config=speculative_config)

llama.cpp runs the EAGLE-4 CPU and phone results in the paper; the EAGLE team also reports a native EAGLE-3 integration.

When speculation stops helping

  • Large batches. Once many requests share each step, the target pass is no longer waiting on memory; every extra verified token costs real compute. The EAGLE-3 abstract's number at batch size 64 is 1.38x, not 6.5x, and the block drafters dip below 1.0x at 512 requests. Serving engines that pack many requests into one step with continuous batching are exactly where adaptive drafting matters.
  • High temperature. A flat target distribution means fewer accepted guesses. In Table 2, EAGLE-4 drops from 3.59x at T = 0 to 2.59x at T = 1, and the built-in MTP head from 2.22x to 1.57x.
  • Text the drafter never saw. A drafter trained on chat, drafting legal contracts or a rare programming language, lands in the cliff on the left of the playground.
  • Time to first token. Speculation speeds up decoding, not prefill, which is already compute-bound. (The phone result above is the exception in the paper, and it comes from other optimizations.)

If you would rather start from the inference basics before tuning any of this, the course has a lesson on speculative decoding and other speedups.

The whole thing on one screen

the drafter, 2022 to 2026 2022 token drafters a small sibling model 2024 EAGLE-1 feature drafter 2024 EAGLE-2 dynamic trees 2025 EAGLE-3 fusion + training-time test 2026 EAGLE 3.1 normalization 2026 EAGLE-4 harness + efficient LM head 1 weights in HBM 54 GB bf16, read every step bottleneck 2 target pass 1 token or 16 cost about the same 3 drafter proposes k tokens as a chain or tree 4 target verifies every candidate in one forward pass 5 accept accepted prefix + correction or bonus 6 commit 1 to k+1 tokens per target pass next round the three knobs acceptance rate how often the target agrees draft cost time per draft step batch size how busy the target already is the drafter, 2022 to 2026 2022 token drafters a small sibling model 2024 EAGLE-1 feature drafter 2024 EAGLE-2 dynamic trees 2025 EAGLE-3 fusion + training-time test 2026 EAGLE 3.1 normalization 2026 EAGLE-4 harness + efficient LM head 1 weights in HBM 54 GB of bf16, read in full every step bottleneck 2 target pass 1 token or 16: about the same cost 3 drafter proposes k tokens, as a chain or a tree 4 target verifies every candidate in one forward pass 5 accept accepted prefix + correction or bonus 6 commit 1 to k+1 tokens per target pass the three knobs acceptance rate how often the target agrees draft cost time per draft step batch size how busy the target already is
The mental model: weights stream from HBM and dominate the step, so the target verifies a chain or tree of guesses almost for free; EAGLE's generations moved each term of the trade-off, and three knobs decide the result.

The whole field is one inequality: the tokens a round gains, divided by the time it spends, must beat one token per target pass. Every generation of EAGLE moved a different term of it. EAGLE-1 made the guesses better by reading the model's features. EAGLE-2 spent the verification budget where the drafter was unsure. EAGLE-3 made the head keep improving with data. And EAGLE-4 measures the denominator on your own hardware, every round, for every batch size.

Your GPU will still spend most of a decoding step waiting for memory. It just no longer waits alone.

Deep dive

The details the first half of the post summarized, one short section each.

Why one run checks every position

A transformer reads all the positions of its input in parallel, and a causal mask in its attention layers stops each position from looking at anything after it. So when the target reads "The cat sat on the mat and fell asleep", the output at each position is exactly its prediction for the next word given only what came before: after "the" it predicts "mat", after "and" it predicts "purred", and so on. This is the same mechanism as prefill, the step where the model reads your prompt in one go, which is why reading a prompt is fast and writing an answer word by word is slow.

The bandwidth math

At batch size 1, one decoding step reads every weight of the model from GPU memory once and multiplies it by the vector of the one token being generated. That is a matrix-vector multiply: about one or two floating point operations per byte read. A modern GPU can do hundreds of operations in the time it takes to move one byte, so the step is limited by memory bandwidth, not by math.

For Qwen3.8-27B in bf16, 27 billion parameters times 2 bytes is 54 GB per step. An H100 SXM moves 3.35 TB/s from its HBM (NVIDIA):

54 GB / 3.35 TB/s = 16.1 ms per token   ->   at most about 62 tokens/s

That is a floor from bandwidth alone, before any KV-cache reads or kernel overhead. Real measurements agree on the order of magnitude. A Blackwell Ultra GPU (the chip in B300 systems) moves 8 TB/s (NVIDIA), which puts the floor at 6.75 ms, about 148 tokens/s; the EAGLE-4 support matrix measures plain Qwen3.8-27B at 103.1 tokens/s in SGLang and 104.0 in vLLM on one B300 (support matrix). AMD's MI355X also has 8 TB/s (AMD) and measures 82 tokens/s for one request (AMD). The gap between floor and measurement is kernel launches, KV-cache traffic and the parts of the step that are not weight reads. Because the weights dominate, running the model on 1 token or on 7 costs nearly the same.

Expected words per round

If each drafted token is accepted with a constant probability a, and the drafter guesses k tokens, the expected number of tokens per target run is (Leviathan et al., Section 3):

E[tokens per target run] = (1 - a^(k+1)) / (1 - a)

With k = 4 and a = 0.8, that is 3.36, the "about 3.4" in the takeaways. With k = 6: a = 0.8 gives 3.95, a = 0.6 gives only 2.43, and a = 0.9 gives 5.22. Push a to 0.95 and you can afford k = 12, for 9.73 tokens per run.

The playground turns that into a speedup with the simplest honest cost model, where D is the time per draft guess and T the time per target run:

time per round     = k x D + T
tokens per round   = (1 - a^(k+1)) / (1 - a)
speedup            = T / (time per round / tokens per round)

The acceptance and resampling rule

Let p be the target's probability distribution over the next token at a position, and q the drafter's. Accept the drafted token x with probability min(1, p(x) / q(x)). If it is rejected, sample a replacement from the positive part of p - q, normalized. Both original papers prove that the resulting tokens are distributed exactly as p (Leviathan et al., appendix; Chen et al., Theorem 1).

For the three-word example from earlier:

tokenp (target)q (drafter)accept if draftedleftover max(0, p - q)
A0.50.2min(1, 0.5 / 0.2) = 10.3
B0.30.5min(1, 0.3 / 0.5) = 0.60
C0.20.3min(1, 0.2 / 0.3) = 0.6670

The leftover normalizes to 1, 0, 0, so every rejection resamples A, which gives the totals 0.5, 0.3 and 0.2 computed earlier: exactly p.

Tree attention masks

To verify a whole draft tree in one run, the target reads every node of the tree as if it were one long sequence and uses an attention mask in which each node may attend to the prompt, to itself and to its own ancestors, and to nothing else. In our tree, "asleep" sees "mat", "and" and "fell"; "purred" sees "mat" and "and" but not "fell" or "asleep". Siblings never see each other, so each branch is scored as if it were the only continuation. EAGLE-2 scores a node by the product of the draft probabilities along its path, expands the most promising nodes, and reranks to pick which ones to send; any prefix of that ranked list is itself a valid tree.

The model behind the numbers

Qwen3.8-27B (Alibaba, released August 14, 2026, Apache 2.0) has 27 billion parameters and no mixture of experts, so all of them are read on every step. It has a 248,320-token vocabulary and a 262,144-token native context. It is not a plain transformer: its 64 layers come in 16 repeating blocks of three Gated DeltaNet layers and one Gated Attention layer, with a hidden size of 5,120. It also ships with its own multi-token prediction (MTP) head, which makes it a good yardstick. For this model, EAGLE-4's one-layer drafter has 0.44B layer parameters per drafted position (EAGLE-4 paper, Table 1).

Frequently asked questions

What is speculative decoding?

A way to make a large language model generate text faster without changing its output. A cheap drafter guesses several next tokens, the big model checks all of them in one forward pass, keeps the longest correct prefix and adds one token of its own. Because decoding is limited by memory bandwidth, checking several tokens costs about the same as generating one.

Does speculative decoding change the model's answers?

No. With greedy decoding the output is identical. With sampling, a rejection-sampling rule (accept a drafted token with probability min(1, p/q), otherwise resample from the normalized positive part of p - q) makes the output distribution exactly the target model's. Engines note that floating point and batching can still cause tiny bit-level differences.

What is EAGLE in speculative decoding?

A family of drafters that predict from the target model's own hidden features instead of from tokens alone. EAGLE-1 drafts at the feature level with a one-layer head, EAGLE-2 builds dynamic draft trees, EAGLE-3 fuses features from several layers and trains with training-time test, and EAGLE-4 adds a harness that sizes the draft per batch size and hardware plus a faster low-rank LM head.

What is the acceptance rate and why does it matter?

The chance that a drafted token matches what the target would have produced. With a constant acceptance rate a and draft length k, one target pass yields (1 - a^(k+1)) / (1 - a) tokens on average: about 3.95 at a = 0.8 and k = 6, but only 2.43 at a = 0.6. The speedup lives or dies on it.

When does speculative decoding stop helping?

At large batch sizes, where the target pass is no longer memory-bound and every extra verified token costs real compute; at high temperature, where acceptance drops; and on text the drafter was not trained for. It also does not speed up the prefill, the time to first token.

Which inference engines support speculative decoding?

vLLM (draft model, n-gram, EAGLE-3, MTP, suffix and more via --speculative-config), SGLang (EAGLE, EAGLE3, DFLASH, STANDALONE, NGRAM via --speculative-algorithm), TensorRT-LLM (Eagle3, MTP, DraftTarget, NGram, PARD, DFlash) and llama.cpp.

References