Your GPU is bored: speculative decoding from first principles to EAGLE-4
Why decoding is bandwidth-bound, how draft-then-verify gives the same answer 3x faster, and how EAGLE-1 to EAGLE-4 learned to guess. With Qwen3.8-27B numbers.
// tags / AI infrastructure
4 posts
Why decoding is bandwidth-bound, how draft-then-verify gives the same answer 3x faster, and how EAGLE-1 to EAGLE-4 learned to guess. With Qwen3.8-27B numbers.
Follow one prompt through llama.cpp on an RTX 5090: how a 16.5 GB Qwen3.8-27B GGUF is memory mapped, copied to VRAM, cached and turned into tokens.
llama.cpp's --moe-cache-mib keeps MoE experts in RAM and caches the hot ones in VRAM: a 94 GB model on a 24 GB RTX 4090, with each MoE flag explained.
One formula explains why some AI runs on a $2 chip and some needs racks of GPUs. A 4-tier map of model size, memory, and hardware, with real numbers.
We use cookies to improve this site. Privacy policy