35 min read by CloudMash
Your GPU is bored: speculative decoding from first principles to EAGLE-4
A small model guesses the next few words, the big model checks them all at once, and you get the same answer in a fraction of the time. Here is how, from the core idea to EAGLE-4.
A trick that lets a language model write several words at a time without changing a single word of its answer, explained slowly with one sentence about a cat, then followed all the way to EAGLE-4.
One word at a time
When a language model answers you, it writes one word at a time. (Strictly, it writes tokens, small chunks of text that are often whole words. In this post you can read "token" as "word".)
To write each word, the model runs its whole network once, start to finish. For a model with tens of billions of internal numbers (its parameters), that is a lot of work for a single word. Then it does it all again for the next word, and the next.
And most next words are boring. After "The cat sat on the", you already know the next word is probably "mat". After "Thank you very", it is "much". The model still runs its entire network to find that out, one word per run.
So here is the question this post is about: if the next few words are that obvious, why not let something cheap guess several of them, and have the big model only check the guesses?
That is speculative decoding. The surprising part, which we will get to, is that checking four guesses takes the big model about as long as writing one word. When the guesses are good, you get several words for the price of one, and the answer comes out exactly the same.
Guess, then check
This is the heart of the idea, so we will take one round slowly, with one example that the rest of the post keeps coming back to.
The two models and the sentence
Two models take part.
The target is the big model, the one whose answers we actually want. In this post it is Qwen3.8-27B, an open model with 27 billion parameters released in August 2026.
The drafter is a much smaller and faster model. Its only job is to guess what the target would write next. It does not have to be right; it only has to be right often.
The text so far is:
The cat sat on the
The drafter guesses four words, one after another: "mat", then "and", then "fell", then "asleep". These are sensible guesses. Cats sit on mats, "mat and" is a natural way to go on, and a cat on a mat might well fall asleep.
But the target, it turns out, would have written "The cat sat on the mat and purred". So the drafter's first two guesses are right and its third is wrong.
One round, step by step
Step 1, guess. The drafter writes its four guesses, one at a time: "mat", "and", "fell", "asleep". It is small, so each guess is quick.
Step 2, check, all at once. The target reads the original text with all four guesses attached, "The cat sat on the mat and fell asleep", in a single run. Here is the key fact: in that one run, the target works out what it would have written at every position along the way, not just at the end. It does exactly the same thing whenever it reads your prompt. (Why one run checks every position is in the deep dive at the end.) So after one run, the target knows that after "the" it would write "mat", after "mat" it would write "and", and after "and" it would write "purred".
Step 3, compare from left to right. "mat" matches what the target wanted: keep it. "and" matches: keep it. "fell" does not: the target wanted "purred". So "fell" is thrown away, and so is "asleep", because it was built on top of a wrong word. In their place goes the target's own word, "purred". That correction costs nothing extra: the target already worked it out during the same run.
The round ends with three new words, "mat and purred", from a single run of the big model. The next round starts from "The cat sat on the mat and purred", and the drafter guesses again.
Here is that same round as a picture. The drafter's four guesses (the candidates) sit on top, and the target's check is below. Green means kept, red marks the first wrong guess, gold is the target's correction, and the dimmed word is the one thrown away.
The same round as a table, one row per position:
| position | target says | drafter said | result |
|---|---|---|---|
| after "the" | mat | mat | keep |
| after "mat" | and | and | keep |
| after "and" | purred | fell | wrong: use "purred" |
| after "fell" | (not needed) | asleep | thrown away: it came after a mistake |
What one round saves
Give the two models round, made-up costs: say one run of the target takes 50 ms and one guess from the drafter takes 5 ms.
Without guessing, the target writes "mat" (50 ms), then "and" (50 ms), then "purred" (50 ms): three words in 150 ms.
With guessing, the drafter makes four guesses (4 x 5 ms = 20 ms) and the target checks them in one run (50 ms): the same three words in 70 ms.
| work done | time | new words | time per word | |
|---|---|---|---|---|
| no guessing | 3 target runs | 150 ms | 3 | 50 ms |
| guessing, this round | 4 guesses + 1 target run | 70 ms | 3 | 23.3 ms (2.14x faster) |
| guessing, all 4 guesses right | 4 guesses + 1 target run | 70 ms | 5 | 14 ms (3.57x faster) |
| guessing, first guess wrong | 4 guesses + 1 target run | 70 ms | 1 | 70 ms |
Three things to notice
- You always get at least one new word per target run. Even if the very first guess is wrong, the target's correction counts. The worst case wastes only the drafter's 20 ms.
- If every guess is right, you get a bonus word. The target's single run also tells it what comes after the last guess. Four right guesses plus that bonus is five words for one run, which is the third row of the table.
- How often the guesses are right decides everything. This is called the acceptance rate. If the drafter is usually right, most rounds keep three or four guesses and the speedup is large. If it is usually wrong, most rounds keep only the correction and the guessing time is wasted. With guesses that are right 80% of the time and four guesses per round, a round produces about 3.4 words on average. (The formula is in the deep dive.)
And one thing that never changes: the words that come out are exactly the words the target would have written on its own. The drafter only proposes; the target decides. A later section shows why this holds even when the target picks its words randomly.
Try it
Now that green and red mean something, play with the two numbers that matter. Acceptance rate is how often a guess is right. Draft length is how many words the drafter guesses per round. The two cost boxes are the time per guess and per target run (5 ms and 50 ms, as above). Leave batch pressure at 1 for now; it comes back in the section on production. Press "run a round" to watch one round play out with random guesses.
// interactive toy model
Speculative decoding playground
Set how often the drafter guesses right and how far ahead it guesses, then run a round and watch the target model check every guess in one pass.
- tokens per target pass
- 3.36
- speedup
- 2.40x
- speculative ms per token
- 20.8 ms
- vanilla ms per token
- 50.0 ms
step = 4 x 5 + 50 x 1.00 = 70.0 ms
context The cat sat on the
Press "run a round": the drafter has proposed 4 tokens, the target will check them all in one pass.
Total: 0 tokens over 0 rounds.
A constant a is a simplification: real engines (EAGLE-2 and later) use the drafter's own confidence for each token, which is why draft trees beat a single chain.
Two things show up quickly. The best number of guesses grows as the guesses get better: guessing far ahead only pays when you guess well. And when guesses are often wrong, the speedup collapses, because you pay for every guess and keep almost none. Real systems are smarter than a fixed hit rate: from EAGLE-2 on, they use how confident the drafter is about each word, which is where this post is heading.
In two sentences: a small model guesses the next few words, and the big model checks all of them in a single run, keeps the ones it agrees with, and puts its own word where the first wrong guess was. You get at least one word per run of the big model and often several, and the text is exactly what the big model would have written alone.
Everything else in this post, including the whole EAGLE family, is about Step 1: making the guesses better, so more of them are kept per round.
Why checking is almost free
Why does checking four guesses cost the target about the same as writing one word? Because of what actually slows a big model down.
To produce anything at all, the model has to read every one of its parameters from the GPU's memory: for Qwen3.8-27B, 27 billion numbers, about 54 GB. Reading them takes far longer than the arithmetic done with them. While a model writes one word at a time, the GPU spends most of each run waiting for numbers to arrive from memory.
Once the numbers are loaded, using them for four words instead of one adds a little arithmetic and almost no time. The extra words ride along in the same transformer run, using compute that was sitting idle anyway. On an H100, reading those 54 GB alone takes about 16 ms per run, however many words are being checked. (The full bandwidth math is in the deep dive.)
The diagram shows the two cases side by side: one word per run and seven words per run, with the same 54 GB of weights flowing through the same pipe from memory to the compute units.
That is the free lunch. The guesses are cheap, and checking them is almost free, so the only question left is how to guess well.
Why the output is exactly the same
When the target always picks its single most likely word, this is easy to see: keep a guess only if it equals the target's choice, and the text is word for word what the target would have written alone.
Most chat models pick words with some randomness, though. Then the rule is a little cleverer. A guess is kept with a probability that depends on how much the target likes that word compared with how much the drafter liked it: if the target likes it at least as much, it is always kept; if less, it is kept only some of the time. When a guess is rejected, the replacement is drawn from the words the drafter gave too little weight to.
A tiny example shows that the bookkeeping works out exactly. Imagine a vocabulary of three words, A, B and C. The target wants them 50%, 30% and 20% of the time. The drafter, slightly off, guesses them 20%, 50% and 30% of the time.
| word | target wants | drafter guesses | chance a guess is kept | gets the rejected share |
|---|---|---|---|---|
| A | 0.5 | 0.2 | 1 | all of it |
| B | 0.3 | 0.5 | 0.6 | none |
| C | 0.2 | 0.3 | 0.667 | none |
A comes out when the drafter guesses A (0.2, always kept) plus every rejection (0.5 x 0.4 + 0.3 x 0.333 = 0.3), so 0.5. B comes out when the drafter guesses B and it is kept: 0.5 x 0.6 = 0.3. C: 0.3 x 0.667 = 0.2. That is exactly what the target wanted. The drafter's mistakes cost speed, never correctness. (The exact rule is in the deep dive.)
The bars show the same example: what the target wants, what the drafter guesses, and how kept guesses plus resampled rejections add up to the target's numbers.
In practice, vLLM calls its sampler "theoretically lossless up to the precision limits of hardware numerics" and warns that floating point and batching effects can still make outputs differ bit for bit (vLLM docs). The math is exact; the hardware is not quite.
The drafter problem
The trick was found in 2022 by two teams independently: Yaniv Leviathan, Matan Kalman and Yossi Matias at Google (arXiv 2211.17192), and Charlie Chen and colleagues at DeepMind (arXiv 2302.01318). Google reported a 2x to 3x speedup on T5-XXL with identical outputs; DeepMind 2x to 2.5x on Chinchilla 70B. In both, the drafter was simply a smaller language model.
That recipe needs a second, smaller model with the same tokenizer, trained separately, that happens to agree with the big one. That is clumsy. A 1B model drafting for a 70B model has never seen the big model's internal state; it guesses from the visible tokens alone. It takes its own memory. It is not free: k draft passes per step. And for many model families there is no small sibling at all.
Making the drafter bigger is the obvious fix, and it mostly does not work: a bigger drafter guesses better but costs more per guess. The EAGLE team made exactly that point in a June 2026 thread, and the EAGLE-4 paper puts numbers on it for our model. The 2026 block drafters for Qwen3.8-27B, DFlash 2 and DSpark, stack 5 layers (about 1.66B and 1.65B layer parameters per drafted position), while EAGLE-4 uses 1 layer (0.44B), spending about 3.8x fewer FLOPs per drafted token (paper, Table 1 and Section 1).
Think back to our sentence. A separate small model guessing "mat", "and", "fell", "asleep" only ever sees the words "The cat sat on the". It has no idea what the target was actually thinking when it read them, which is why its third guess went wrong. So the real question is different: what if the drafter could read the big model's mind instead of its mouth?
EAGLE-1: draft from the model's thoughts
EAGLE (arXiv 2401.15077, ICML 2024) starts from two observations.
First, predicting the target's next feature (the hidden state just before the LM head, from its second-to-top layer) is much easier than predicting its next token. Features change smoothly from step to step; tokens jump around.
Second, features carry a built-in uncertainty: the next feature depends on which token was actually sampled, and sampling is random. EAGLE's fix is to feed the drafter the token sequence shifted one step ahead alongside the features, so it always knows which token was picked.
The drafter is a single lightweight decoder layer, the EAGLE head. It takes the current feature plus the embedding of the next token and predicts the next feature; the target's own frozen LM head turns that predicted feature into a token distribution. The target's weights are never touched; only the small head is trained, on the target's own features.
Back to our sentence. When the target finishes reading "The cat sat on the", its last-layer feature already leans strongly toward "mat". The EAGLE head reads that feature plus the token "mat" and predicts the feature for the next position, where "and" is the likely word, without running the 27-billion-parameter stack again. That is why its guesses are good where a small sibling model's guesses are mediocre.
The diagram shows exactly that: the target's tall stack of layers, a tap on its second-to-top layer feeding the small EAGLE head beside it, the token "mat" joining from below, and the target's own LM head on top turning the head's prediction into the next word, "and".
The paper reports a 2.7x to 3.5x latency speedup on LLaMA2-Chat 70B and doubled throughput. The repository's 13B demo puts EAGLE-1 at 3x over plain decoding, 2x over Lookahead and 1.6x over Medusa.
EAGLE-2: dynamic draft trees
A chain of guesses has a weakness: one wrong token kills everything after it. Look at our example. After "and", a reasonable drafter is torn between "fell" and "purred". A chain must bet on one, and our chain bet on "fell". A tree can send both.
EAGLE-1 already used trees, but with one fixed shape for every context. EAGLE-2 (arXiv 2406.16858, EMNLP 2024) noticed that the drafter's confidence (the probability it assigns to its own guess) is well calibrated: it predicts how likely the target is to accept. So the tree can be shaped per context, in two phases:
- Expand: grow the tree by extending the most promising nodes, where a node's score is the product of the draft probabilities along its path.
- Rerank: pick the top nodes overall and send those for verification. Any prefix of that ranked list is itself a valid tree.
The target still checks everything in one run. It reads the whole tree at once, and a mask makes sure each guess only looks at the words on its own branch, so "fell" and "purred" are checked side by side, as if each were the only way to continue. (How the tree attention mask works is in the deep dive.)
The diagram shows our sentence as a tree: "mat" and "and" as a shared trunk, then a split into "fell" (with "asleep" after it) and "purred". Beside the tree is the mask, a grid showing which guesses each guess is allowed to look at.
With a tree, the third guess is no longer a rejection: the target picks "purred", that branch is accepted, and the round commits "mat and purred" as three accepted drafts plus a bonus token from the same pass. EAGLE-2 reports 3.05x to 4.26x across three model series and six tasks, 20% to 40% faster than EAGLE-1, with the same drafter and no retraining: the gain is entirely in how the tree is built.
EAGLE-3 and EAGLE 3.1
EAGLE-1 and 2 trained the head to reproduce the target's next feature. That objective was also a ceiling: forcing the head to copy a feature limits how much it improves with more data. EAGLE-3 (arXiv 2503.01840, NeurIPS 2025) made three changes.
- Predict tokens directly. Drop the feature-regression loss and train the head on the thing that matters, the next token.
- Multi-layer feature fusion. Instead of one tap at the second-to-top layer, concatenate low-, mid- and high-level features from several target layers, then project them down. Early layers know about syntax and copying, late layers about meaning; the head gets all of it.
- Training-time test. At inference, the head's later draft steps consume the head's own previous outputs, not fresh target features. EAGLE-1 and 2 only ever trained on real target features, so they were tested on inputs they never saw in training. EAGLE-3 simulates multi-step drafting during training, so training matches use.
That last change is the one that lets acceptance keep growing as you add training data.
For our sentence, the difference is in the third guess. EAGLE-1 had one view of the target's thinking after "The cat sat on the mat and"; EAGLE-3 sees several layers of it at once and was trained on chains of its own guesses, so it is better placed to rank "purred" above "fell" in the first place.
The diagram shows the three taps at different heights of the target's stack merging into one input for the head, and a dotted loop where the head is trained on its own previous outputs.
The abstract reports up to 6.5x speedup, about 1.4x over EAGLE-2, and 1.38x throughput at batch size 64 in SGLang. SGLang's docs show the progression on one H100 with Llama-3.1-8B-Instruct on MT-Bench: 158.34 tokens/s without speculation, 244.10 with EAGLE-2, 373.25 with EAGLE-3 (SGLang docs).
EAGLE 3.1 (May 2026, the EAGLE team with vLLM and TorchSpec, vLLM blog) fixed something subtle the team calls attention drift: as the draft gets deeper, the head attends less to the important early tokens and more to its own guesses, partly because the higher-layer features dominate the fused input and the hidden state grows in magnitude step after step. Two small changes, a normalization of each target feature before fusion and post-normalized hidden states fed into the next step, keep the feedback loop stable. The post reports long-context acceptance length up to 2x that of EAGLE-3, and on Kimi-K2.6 NVFP4 with vLLM on GB200 (TP4), 2.03x per-user throughput at concurrency 1, 1.71x at 4 and 1.66x at 16. The work, described in the paper "Attention drift" (arXiv 2605.09992), was accepted to NeurIPS 2026. The team announced it on X.
EAGLE-4: one drafter for every batch size and every chip
Every method so far has a hidden assumption: that there is one right amount of drafting. There is not. At batch size 1 the GPU is idle, so you should draft deep and wide. At 512 concurrent requests the target pass is compute-bound, every extra verified token costs real time, and the right answer is a token or two. Drafters with a fixed block size cannot adapt, and the EAGLE-4 paper shows what happens: at 512 concurrent requests, the 2026 block drafters fall below plain decoding (0.78x and 0.67x).
EAGLE-4 (Eagle AI, released October 20, 2026; paper, repo) attacks that with three ideas.
EAGLE-Harness: measure the hardware, then decide
At startup, the harness profiles two costs on the machine it is running on: the time of a draft step, and the time to verify a given number of tokens, indexed by batch size and context length. It caches those tables. During decoding, every decision is a single comparison: extend the tree one more level, or verify one more token, only if the expected gain in accepted tokens is worth the measured time:
deltaA / deltaC >= tau
The threshold tau rises with depth, because deep estimates are less reliable (Section 3.1, Algorithm 1). Under light load the tree may branch up to 8 ways per node; under heavy load at most 4. The paper's configurations show the range: depth 12, width 4 and 16 verified tokens at concurrency 1, versus a 15-deep chain of width 1 at concurrency 512. Same weights: wide, branching trees for one user, a single narrow chain when the server is full. No retraining, only table lookups and scores the drafter already computes.
For our sentence, that means one person typing "The cat sat on the" might get a wide tree that tries both "fell" and "purred" after "and", while on a server handling 512 conversations the same drafter sends one guess per position and spends nothing on branches that will probably be thrown away.
The diagram shows the loop: cost tables measured once, then for every step the single comparison that decides whether to guess further, how many guesses to send for checking, and the check itself. The inset compares the tree built for one request with the one built for 512.
The Efficient LM Head
Once the drafter is one layer, the slowest part of it is the vocabulary. Scoring all 248,320 entries of Qwen3.8-27B's vocabulary was 65.4% of a 574 microsecond draft step (paper, Figure 2). EAGLE-4 adds a low-rank head (rank 128 to 512) that only shortlists K candidates (128, 256 or 512) from the full vocabulary, then rescores exactly those K rows with the real LM head. The work drops from O(V d) to O(d r + V r + K d), and the draft step from 574 microseconds to 326 (rank 512) and 255 (rank 128), cuts of 43.2% and 55.6%.
The trick is training the two low-rank factors for retrieval. A plain SVD initialization at rank 128 lost 0.48 tokens of acceptance length per verification; the trained head keeps the full head's top choice in its shortlist 98.2% of the time (91.5% for SVD) and shows no acceptance loss. Draft time per verification cycle halves, from 9.1 ms to 4.8 ms at batch size 1. The drafter also uses sliding-window attention for long contexts, an append-only cache, and a feed-forward layer that mirrors the target, dense or MoE.
The results
On Qwen3.8-27B with four B300 GPUs (TP4, SPEED-Bench, paper Table 2):
| method | 1 request, T = 0 | 1 request, T = 1 | 512 requests, T = 0 |
|---|---|---|---|
| plain decoding | 240.5 tok/s | 234.9 | 21,203 |
| built-in MTP head | 534.5 (2.22x) | 369.1 (1.57x) | 21,518 (1.01x) |
| DSpark | 618.6 (2.57x) | 519.4 (2.21x) | 14,229 (0.67x) |
| DFlash 2 | 672.6 (2.80x) | 560.3 (2.39x) | 16,641 (0.78x) |
| EAGLE-4 | 863.8 (3.59x) | 608.6 (2.59x) | 28,340 (1.34x) |
On HumanEval at one request it reaches 929 tok/s (Figure 3). Vendors that measured it on their own stacks report similar shapes: AMD sees about 4.7 tokens accepted per check against about 2.9 for the built-in MTP head (AMD), and LightSeek reports 985 tok/s greedy on TokenSpeed with the same four B300s (LightSeek). The team's own launch post quotes a higher single-request number with TokenSpeed; the paper's figures are the ones above.
Then the part that makes this fun. The same algorithm runs well away from the data center:
- One AMD MI350X (paper Table 4): 75.27 to 262.89 tok/s at one request (3.49x), still 1.70x at 32. AMD shipped day-0 support on MI355X: 82 to 289 tok/s in SGLang and 309 on its ATOM engine for one request.
- A 4-vCPU cloud CPU (AWS c5.xlarge, 8 GB, llama.cpp, 4-bit Gemma-4-E2B-it, Table 5): 2.28x to 2.92x, 2.71x on average, where EAGLE-3 managed 1.25x to 1.65x on the same box. The whole draft head is 134 MB; total extra memory 0.27 GB.
- A phone (Motorola Razr 70 Ultra, Snapdragon 8 Elite, unmodified and not rooted, llama.cpp on 6 threads, Table 6): HumanEval from 22.8 to 82.6 tok/s (3.62x), long-form writing 23.3 to 74.7 (3.21x). Energy per generated token fell from 356 mJ to 166 mJ, measured at the USB input. A generation that took 43 seconds takes 14.
The algorithm that feeds four B300s also halves the battery cost of a token on a phone.
If you were waiting for a speculative-decoding chip: as of November 2026 no primary source announces one from Eagle AI. The more interesting news is the opposite. The drafter now treats the chip as a parameter: NVIDIA sm_80 and newer, AMD CDNA2 to CDNA4, x86 CPUs, an Arm phone.
Go deeperlicense, EAGLE-5, the wider EAGLE footprint
EAGLE-4 is source-available, not OSI open source: commercial use above a 20 million US dollar aggregate-revenue threshold, or any competing offering, needs a separate license (README). The earlier EAGLE-1, 2 and 3 code (SafeAILab/EAGLE) is Apache 2.0. GMI Cloud has announced a closed, hosted successor called EAGLE-5 (GMI Cloud).
EAGLE ideas also run well outside the EAGLE repos. NVIDIA's record of over 1,000 tokens per second per user on Llama 4 Maverick on one DGX B200 used a drafter trained with EAGLE-3 techniques; speculation alone gave about 2.06x at draft length 3 (NVIDIA). On Jetson AGX Thor, EAGLE-3 is the biggest piece of a 7x cumulative gain on Llama 3.3 70B, and 2.5x on its own at W4A16 and one request (NVIDIA).
The wider family
EAGLE is one branch of a busy tree. Each of these solves a different piece of the problem.
Medusa (arXiv 2401.10774) skips the separate drafter: several extra LM heads sit on the target's last hidden state, each predicting the token k steps ahead, and their candidates are verified with tree attention. Medusa-1 trains only the heads (over 2.2x); Medusa-2 fine-tunes the backbone too (2.3x to 3.6x), so it is no longer strictly lossless. Hydra (arXiv 2402.05109) makes those heads depend on each other's guesses, for up to 1.31x over Medusa.
SpecInfer (arXiv 2305.09781) uses small models to build a token tree and verifies it in one tree-parallel pass: 1.5x to 2.8x for distributed serving, 2.6x to 3.5x with offloading.
Lookahead decoding (arXiv 2402.02057) needs no drafter at all: Jacobi iteration guesses several future positions at once, an n-gram pool collects what it finds, and the same pass verifies them. Up to 1.8x on MT-Bench and up to 4x on multi-GPU code completion.
Prompt lookup (repo) is the cheapest drafter possible: find the last few tokens earlier in the prompt and propose what followed them. 2.4x on average for summarization and document QA with Mistral-7B-Instruct on an A100. It shines when the output repeats the input: in a code edit or a summary, the next words often already exist in the prompt. That is also why wild copy-workload anecdotes circulate online: when the output is a copy of the input, nearly every guess is accepted.
Self-speculative decoding drafts with the model itself. Draft & Verify (arXiv 2309.08168) skips layers chosen by Bayesian optimization, with no training, for up to 1.99x. LayerSkip (arXiv 2404.16710) trains the model with an early-exit loss so its early layers can draft: up to 2.16x on summarization, 1.82x on code.
ReDrafter from Apple (arXiv 2403.09919) uses a small RNN head with beam search: up to 2.8x for Vicuna on an H100, 2.3x with MLX on Apple Silicon.
Multi-token prediction heads turn drafting into a training feature. DeepSeek-V3 (arXiv 2412.19437) trains one extra MTP module and keeps it at inference: the second token is accepted 85% to 90% of the time, for 1.8x tokens per second. The EAGLE-4 paper points out that mainstream MTP heads follow EAGLE's design of concatenating the target's hidden state with the token embedding; Qwen3.8-27B's own head is the 534.5 tok/s row in the table above.
| method | drafter | extra training | lossless | headline number (setting) | best for |
|---|---|---|---|---|---|
| Leviathan et al. | small model | none, needs a sibling | yes | 2x to 3x (T5-XXL) | any model with a small sibling |
| Chen et al. | small model | none, needs a sibling | yes | 2x to 2.5x (Chinchilla 70B) | distributed serving |
| SpecInfer | small models, tree | small drafters | yes | 1.5x to 2.8x distributed | distributed and offloaded serving |
| Medusa-1 / Medusa-2 | extra heads | heads (M2 also backbone) | M1 yes, M2 no | over 2.2x / 2.3x to 3.6x | one model, batch 1 |
| Hydra | dependent heads | heads | yes | 1.31x over Medusa | Medusa upgrade |
| Lookahead | none (Jacobi) | none | yes | up to 1.8x (MT-Bench) | no drafter, spare FLOPs |
| Prompt lookup | prompt n-grams | none | yes | 2.4x avg (Mistral-7B, A100) | RAG, summaries, code edits |
| Draft & Verify | skipped layers | none | yes | up to 1.99x (LLaMA-2) | zero extra memory |
| LayerSkip | early exit | retrain target | yes | up to 2.16x (CNN/DM) | one model, tight memory |
| ReDrafter | RNN head | small head | yes | up to 2.8x (Vicuna, H100) | NVIDIA and Apple Silicon |
| DeepSeek-V3 MTP | built-in head | in pretraining | yes | 1.8x TPS | models trained with MTP |
| EAGLE-1 | feature head | small head | yes | 2.7x to 3.5x (LLaMA2-Chat 70B) | chat, code, low batch |
| EAGLE-2 | same head, dynamic tree | none beyond EAGLE-1 | yes | 3.05x to 4.26x | context-dependent text |
| EAGLE-3 | fused-feature head | small head, more data | yes | up to 6.5x | chat and reasoning |
| DFlash 2 | 5-layer block diffusion | yes | yes | 2.80x (Qwen3.8-27B, 4x B300) | batch 1, long blocks |
| DSpark | 5-layer semi-autoregressive | yes | yes | 2.57x (Qwen3.8-27B, 4x B300) | batch 1, production schedulers |
| EAGLE-4 | 1-layer head + harness | small head | yes | 3.59x at 1 request, 1.34x at 512 | every batch size and chip |
The last three rows are from the EAGLE-4 paper's Table 2 on Qwen3.8-27B; the rest are each paper's own headline.
The block drafters of 2026
The strongest rivals to EAGLE this year do not draft one token at a time. They draft a whole block in one pass.
DFlash (arXiv 2602.06036, z-lab/dflash, MIT) makes the drafter a small block-diffusion model that produces a whole block of tokens at once, conditioned on the target's features; its abstract reports over 6x lossless acceleration and up to 2.5x the speedup of EAGLE-3. DFlash 2 (Inco AI, August 2026) adds a path selector that walks one coherent path through the top 16 candidates per position, plus a small dynamic convolution, for about 1.3% extra latency; Inco reports 2.7x to 3.4x on Qwen3.8-27B in SGLang at batch size 1. Its author announced a DFlash 2 drafter for Qwen3.8-Flash-Next in October.
DSpark (arXiv 2607.05147, DeepSeek-AI and Peking University, code at deepseek-ai/DeepSpec) pairs a parallel DFlash-style backbone with a light sequential "Markov head" for dependencies inside the block, and schedules verification per request from estimated survival probabilities and the engine's current load. In DeepSeek-V4 production serving it makes per-user generation 60% to 85% faster than an MTP-1 baseline. People are running it in surprising places: Zhihao Jia reports Qwen3.8-27B at over 200 tokens per second per user on a single Apple M5 Max with megakernels plus DSpark, his own measurement.
The upside of block drafting is that drafting time does not grow with block length. The downside is the EAGLE-4 paper's argument (Section 1.1): a block drafter has paid for the whole block before it learns which tokens were useless, while an autoregressive drafter can stop before paying for the next step. At large batch the best block shrinks to one or two tokens, where parallel drafting has no advantage left. That is the 0.67x and 0.78x in the table.
Go deeperHASS, GliDe, FR-Spec, PARD, SuffixDecoding, trainers and benchmarks
- HASS (arXiv 2408.15766, ICLR 2025): trains EAGLE-style heads with harmonized objective distillation and context alignment; 2.81x to 4.05x, 8% to 20% over EAGLE-2.
- GliDe with a CaPE (arXiv 2402.02082): the drafter reuses the target's KV cache, and confidence adds extra candidates; up to 2.17x, 2.61x with CaPE.
- FR-Spec (arXiv 2502.14856): restricts the drafter's LM head to a frequency-ranked vocabulary subset, cutting LM-head compute 75%, 1.12x over EAGLE-2. It is the idea EAGLE-4's Efficient LM Head replaces with an exact rescoring.
- PARD (AMD, arXiv 2504.18583): one target-independent parallel draft model per model family; up to 3.67x on Llama-3.1-8B in vLLM.
- SuffixDecoding (Snowflake, arXiv 2411.04975): model-free suffix trees over prompts and earlier outputs; up to 5.3x on SWE-Bench and text-to-SQL agent workloads.
- Trainers: SpecForge (SGLang's MIT trainer for EAGLE3, EAGLE 3.1, DFlash, DFlash 2 and DSpark drafters) and TorchSpec (LightSeek, MIT, disaggregated training; LightSeek says it trained the EAGLE-4 drafters).
- Benchmarks: Spec-Bench (2024; EAGLE-3 at 3.02x with 5.71 mean accepted tokens on Vicuna-13B on an A100) and SPEED-Bench (NVIDIA, arXiv 2604.09557), which the 2026 papers use.
In production
All the major engines speak speculation now.
vLLM takes a --speculative-config JSON whose method includes draft_model, ngram, eagle3, mtp, suffix, dflash and more (docs). The zero-download way to try it is n-gram drafting from the prompt:
vllm serve <target-model> \
--speculative-config '{
"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_min": 2,
"prompt_lookup_max": 5
}'
SGLang uses --speculative-algorithm with EAGLE, EAGLE3, DFLASH, STANDALONE, NGRAM and UNO, and three knobs that map straight onto this post: --speculative-num-steps (depth), --speculative-eagle-topk (branching) and --speculative-num-draft-tokens (how many tokens to verify) (docs). The docs' EAGLE example:
python3 -m sglang.launch_server \
--model meta-llama/Llama-2-7b-chat-hf \
--speculative-algorithm EAGLE \
--speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
--speculative-num-steps 3 \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 16 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
EAGLE-4 plugs into SGLang, vLLM and TokenSpeed through the eagle-harness package; for our model the README's short form is:
sglang serve --model-path Qwen/Qwen3.8-27B \
--speculative-algorithm EAGLE_HARNESS \
--speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4
TensorRT-LLM (PyTorch backend) has Eagle3DecodingConfig, MTPDecodingConfig, DraftTarget, NGram (deprecated), PARD, DFlash and SA (docs). The MTP path for a model with native MTP modules is three lines:
from tensorrt_llm.llmapi import MTPDecodingConfig
speculative_config = MTPDecodingConfig(max_draft_len=3)
llm = LLM("/path/to/deepseek_model", speculative_config=speculative_config)
llama.cpp runs the EAGLE-4 CPU and phone results in the paper; the EAGLE team also reports a native EAGLE-3 integration.
When speculation stops helping
- Large batches. Once many requests share each step, the target pass is no longer waiting on memory; every extra verified token costs real compute. The EAGLE-3 abstract's number at batch size 64 is 1.38x, not 6.5x, and the block drafters dip below 1.0x at 512 requests. Serving engines that pack many requests into one step with continuous batching are exactly where adaptive drafting matters.
- High temperature. A flat target distribution means fewer accepted guesses. In Table 2, EAGLE-4 drops from 3.59x at T = 0 to 2.59x at T = 1, and the built-in MTP head from 2.22x to 1.57x.
- Text the drafter never saw. A drafter trained on chat, drafting legal contracts or a rare programming language, lands in the cliff on the left of the playground.
- Time to first token. Speculation speeds up decoding, not prefill, which is already compute-bound. (The phone result above is the exception in the paper, and it comes from other optimizations.)
If you would rather start from the inference basics before tuning any of this, the course has a lesson on speculative decoding and other speedups.
The whole thing on one screen
The whole field is one inequality: the tokens a round gains, divided by the time it spends, must beat one token per target pass. Every generation of EAGLE moved a different term of it. EAGLE-1 made the guesses better by reading the model's features. EAGLE-2 spent the verification budget where the drafter was unsure. EAGLE-3 made the head keep improving with data. And EAGLE-4 measures the denominator on your own hardware, every round, for every batch size.
Your GPU will still spend most of a decoding step waiting for memory. It just no longer waits alone.
Deep dive
The details the first half of the post summarized, one short section each.
Why one run checks every position
A transformer reads all the positions of its input in parallel, and a causal mask in its attention layers stops each position from looking at anything after it. So when the target reads "The cat sat on the mat and fell asleep", the output at each position is exactly its prediction for the next word given only what came before: after "the" it predicts "mat", after "and" it predicts "purred", and so on. This is the same mechanism as prefill, the step where the model reads your prompt in one go, which is why reading a prompt is fast and writing an answer word by word is slow.
The bandwidth math
At batch size 1, one decoding step reads every weight of the model from GPU memory once and multiplies it by the vector of the one token being generated. That is a matrix-vector multiply: about one or two floating point operations per byte read. A modern GPU can do hundreds of operations in the time it takes to move one byte, so the step is limited by memory bandwidth, not by math.
For Qwen3.8-27B in bf16, 27 billion parameters times 2 bytes is 54 GB per step. An H100 SXM moves 3.35 TB/s from its HBM (NVIDIA):
54 GB / 3.35 TB/s = 16.1 ms per token -> at most about 62 tokens/s
That is a floor from bandwidth alone, before any KV-cache reads or kernel overhead. Real measurements agree on the order of magnitude. A Blackwell Ultra GPU (the chip in B300 systems) moves 8 TB/s (NVIDIA), which puts the floor at 6.75 ms, about 148 tokens/s; the EAGLE-4 support matrix measures plain Qwen3.8-27B at 103.1 tokens/s in SGLang and 104.0 in vLLM on one B300 (support matrix). AMD's MI355X also has 8 TB/s (AMD) and measures 82 tokens/s for one request (AMD). The gap between floor and measurement is kernel launches, KV-cache traffic and the parts of the step that are not weight reads. Because the weights dominate, running the model on 1 token or on 7 costs nearly the same.
Expected words per round
If each drafted token is accepted with a constant probability a, and the drafter guesses k tokens, the expected number of tokens per target run is (Leviathan et al., Section 3):
E[tokens per target run] = (1 - a^(k+1)) / (1 - a)
With k = 4 and a = 0.8, that is 3.36, the "about 3.4" in the takeaways. With k = 6: a = 0.8 gives 3.95, a = 0.6 gives only 2.43, and a = 0.9 gives 5.22. Push a to 0.95 and you can afford k = 12, for 9.73 tokens per run.
The playground turns that into a speedup with the simplest honest cost model, where D is the time per draft guess and T the time per target run:
time per round = k x D + T
tokens per round = (1 - a^(k+1)) / (1 - a)
speedup = T / (time per round / tokens per round)
The acceptance and resampling rule
Let p be the target's probability distribution over the next token at a position, and q the drafter's. Accept the drafted token x with probability min(1, p(x) / q(x)). If it is rejected, sample a replacement from the positive part of p - q, normalized. Both original papers prove that the resulting tokens are distributed exactly as p (Leviathan et al., appendix; Chen et al., Theorem 1).
For the three-word example from earlier:
| token | p (target) | q (drafter) | accept if drafted | leftover max(0, p - q) |
|---|---|---|---|---|
| A | 0.5 | 0.2 | min(1, 0.5 / 0.2) = 1 | 0.3 |
| B | 0.3 | 0.5 | min(1, 0.3 / 0.5) = 0.6 | 0 |
| C | 0.2 | 0.3 | min(1, 0.2 / 0.3) = 0.667 | 0 |
The leftover normalizes to 1, 0, 0, so every rejection resamples A, which gives the totals 0.5, 0.3 and 0.2 computed earlier: exactly p.
Tree attention masks
To verify a whole draft tree in one run, the target reads every node of the tree as if it were one long sequence and uses an attention mask in which each node may attend to the prompt, to itself and to its own ancestors, and to nothing else. In our tree, "asleep" sees "mat", "and" and "fell"; "purred" sees "mat" and "and" but not "fell" or "asleep". Siblings never see each other, so each branch is scored as if it were the only continuation. EAGLE-2 scores a node by the product of the draft probabilities along its path, expands the most promising nodes, and reranks to pick which ones to send; any prefix of that ranked list is itself a valid tree.
The model behind the numbers
Qwen3.8-27B (Alibaba, released August 14, 2026, Apache 2.0) has 27 billion parameters and no mixture of experts, so all of them are read on every step. It has a 248,320-token vocabulary and a 262,144-token native context. It is not a plain transformer: its 64 layers come in 16 repeating blocks of three Gated DeltaNet layers and one Gated Attention layer, with a hidden size of 5,120. It also ships with its own multi-token prediction (MTP) head, which makes it a good yardstick. For this model, EAGLE-4's one-layer drafter has 0.44B layer parameters per drafted position (EAGLE-4 paper, Table 1).
Frequently asked questions
What is speculative decoding?
A way to make a large language model generate text faster without changing its output. A cheap drafter guesses several next tokens, the big model checks all of them in one forward pass, keeps the longest correct prefix and adds one token of its own. Because decoding is limited by memory bandwidth, checking several tokens costs about the same as generating one.
Does speculative decoding change the model's answers?
No. With greedy decoding the output is identical. With sampling, a rejection-sampling rule (accept a drafted token with probability min(1, p/q), otherwise resample from the normalized positive part of p - q) makes the output distribution exactly the target model's. Engines note that floating point and batching can still cause tiny bit-level differences.
What is EAGLE in speculative decoding?
A family of drafters that predict from the target model's own hidden features instead of from tokens alone. EAGLE-1 drafts at the feature level with a one-layer head, EAGLE-2 builds dynamic draft trees, EAGLE-3 fuses features from several layers and trains with training-time test, and EAGLE-4 adds a harness that sizes the draft per batch size and hardware plus a faster low-rank LM head.
What is the acceptance rate and why does it matter?
The chance that a drafted token matches what the target would have produced. With a constant acceptance rate a and draft length k, one target pass yields (1 - a^(k+1)) / (1 - a) tokens on average: about 3.95 at a = 0.8 and k = 6, but only 2.43 at a = 0.6. The speedup lives or dies on it.
When does speculative decoding stop helping?
At large batch sizes, where the target pass is no longer memory-bound and every extra verified token costs real compute; at high temperature, where acceptance drops; and on text the drafter was not trained for. It also does not speed up the prefill, the time to first token.
Which inference engines support speculative decoding?
vLLM (draft model, n-gram, EAGLE-3, MTP, suffix and more via --speculative-config), SGLang (EAGLE, EAGLE3, DFLASH, STANDALONE, NGRAM via --speculative-algorithm), TensorRT-LLM (Eagle3, MTP, DraftTarget, NGram, PARD, DFlash) and llama.cpp.
References
- Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. arXiv 2211.17192
- Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv 2302.01318
- Li, Wei, Zhang, Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv 2401.15077, code SafeAILab/EAGLE
- EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv 2406.16858
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv 2503.01840
- EAGLE 3.1 on the vLLM blog; Eldenk et al. Attention drift. arXiv 2605.09992
- EAGLE-4 Team. EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware. Paper (PDF), repo, support matrix
- AMD. EAGLE-4 on AMD Instinct GPUs from day 0. amd.com
- LightSeek. EAGLE-4 training and serving with TorchSpec and TokenSpeed. lightseek.org
- Medusa. arXiv 2401.10774; Hydra. arXiv 2402.05109
- SpecInfer. arXiv 2305.09781
- Lookahead decoding. arXiv 2402.02057
- Prompt lookup decoding. GitHub
- Draft & Verify. arXiv 2309.08168; LayerSkip. arXiv 2404.16710
- ReDrafter. arXiv 2403.09919
- DeepSeek-V3 Technical Report. arXiv 2412.19437
- DFlash. arXiv 2602.06036; DFlash 2. Inco AI blog
- DSpark. arXiv 2607.05147
- HASS arXiv 2408.15766; GliDe with a CaPE arXiv 2402.02082; FR-Spec arXiv 2502.14856; PARD arXiv 2504.18583; SuffixDecoding arXiv 2411.04975
- Speculative decoding survey and Spec-Bench. arXiv 2401.07851, GitHub; SPEED-Bench. arXiv 2604.09557
- Engine docs: vLLM, SGLang, TensorRT-LLM
- Hardware: NVIDIA H100, NVIDIA Blackwell Ultra, AMD MI355X
- Target model: Qwen3.8-27B model card