Skip to content

// tags / AI infrastructure

AI infrastructure

4 posts

20 min read + video

Your GPU holds 24 GB. This model is 94 GB. It still runs.

llama.cpp's --moe-cache-mib keeps MoE experts in RAM and caches the hot ones in VRAM: a 94 GB model on a 24 GB RTX 4090, with each MoE flag explained.

AI infrastructureLLM inferenceGPUsinternals