Prefill vs Decode
Operational split in LLM serving between processing a block of input tokens in parallel and generating output tokens sequentially one at a time.
Key points
- Decode processes one next token per sequence and is heavily exposed to memory bandwidth because the model repeatedly fetches weights and prior KV-cache data [src-042].
- Prefill processes many input tokens in a pass, so memory costs can be divided over more tokens and the operation is more likely to be compute-limited [src-042].
- The common API pattern where output tokens cost several times more than input tokens is consistent with decode being more memory-bandwidth constrained than prefill [src-042].
- Tool calls, user messages, and file reads create new prefill segments inside a chat or agent session [src-042].
- Understanding this split helps explain why prompt caching and cache hits can reduce price and latency: they avoid rematerializing already processed context [src-042].
Related concepts
Source references
- [src-042] Dwarkesh Patel — "How GPT, Claude, and Gemini are actually trained and served – Reiner Pope" (2026-04-29)
Recommended next
Keep reading from this thread
From 500 indexed pages and articles.
- Wiki concept KV Cache Stored key/value representations of prior tokens used during autoregressive transformer decoding so each new token can attend to the previous context without recomputing every Related by decode
- Wiki concept KV Cache Tiering Serving strategy for deciding whether to keep KV-cache state in fast memory, move it to slower storage, or delete and rematerialize it later Related by 042
- Wiki concept Roofline Analysis for LLM Serving Back-of-the-envelope method for estimating LLM serving latency by comparing compute time against memory-fetch time. Related by 042