KV Cache Tiering

Serving strategy for deciding whether to keep KV-cache state in fast memory, move it to slower storage, or delete and rematerialize it later.

Key points

  • Pope distinguishes between rematerializing KV cache from token IDs and storing previously computed KV cache in memory [src-042].
  • Rematerialization saves storage but requires another forward pass through the model [src-042].
  • KV cache can be stored in different memory tiers such as HBM, host DDR, flash, or slower disks, each with different holding costs and retrieval costs [src-042].
  • Cache pricing durations, such as short-lived versus hour-long cache writes, can hint at which memory tier a provider is using [src-042].
  • The right tier balances hold time against retrieval time: short-lived caches justify faster memory, while long-lived caches may move to cheaper slower tiers [src-042].

Related concepts

Source references

  • [src-042] Dwarkesh Patel — "How GPT, Claude, and Gemini are actually trained and served – Reiner Pope" (2026-04-29)

Robin Cartier perspective

This page is part of Robin Cartier's working AI knowledge graph: a practical research layer for production AI, recommendation systems, experimentation, GEO, and agentic web readiness.

The useful next step is to connect this concept back to applied product leadership and operating models.

Recommended next

Keep reading from this thread

From 500 indexed pages and articles.

  1. Wiki concept KV Cache Stored key/value representations of prior tokens used during autoregressive transformer decoding so each new token can attend to the previous context without recomputing every Related by cache
  2. Wiki concept Prefill vs Decode Operational split in LLM serving between processing a block of input tokens in parallel and generating output tokens sequentially one at a time Related by tiering
  3. Wiki concept Prompt Caching for Agents The practice of arranging stable instructions, policies, and tool schemas so providers can reuse repeated prompt prefixes Related by cache