Roofline Analysis for LLM Serving

Back-of-the-envelope method for estimating LLM serving latency by comparing compute time against memory-fetch time.

Key points

  • Pope models decode time as at least the maximum of compute time and memory time [src-042].
  • Compute time scales with batch size and active parameters divided by chip FLOPs [src-042].
  • Memory time includes weight fetches over total parameters plus KV-cache fetches over batch size, context length, and bytes per token, divided by memory bandwidth [src-042].
  • This simple model explains why serving can shift between compute-bound and memory-bound regimes as batch size or context length changes [src-042].
  • The model also explains why the "Goldilocks" context length matters: beyond the balance point, dense attention makes KV-cache memory bandwidth dominate [src-042].

Related concepts

Source references

  • [src-042] Dwarkesh Patel — "How GPT, Claude, and Gemini are actually trained and served – Reiner Pope" (2026-04-29)

Robin Cartier perspective

This page is part of Robin Cartier's working AI knowledge graph: a practical research layer for production AI, recommendation systems, experimentation, GEO, and agentic web readiness.

The useful next step is to connect this concept back to applied product leadership and operating models.

Recommended next

Keep reading from this thread

From 500 indexed pages and articles.

  1. Wiki concept LLM Serving Batching Serving optimization where multiple user sequences are decoded together so the cost of reading model weights is shared across many generated tokens. Related by serving
  2. Wiki concept Memory Wall for Long Context Infrastructure constraint where extending LLM context length is limited more by KV-cache memory bandwidth and capacity than by raw compute. Related by 042
  3. Wiki concept Mixture-of-Experts Serving Serving architecture for sparse models where a router sends each token to a subset of expert MLPs, reducing active compute while increasing total parameters Related by serving