Mixture-of-Experts Serving

Serving architecture for sparse models where a router sends each token to a subset of expert MLPs, reducing active compute while increasing total parameters and communication complexity.

Key points

• Pope uses DeepSeek-style sparse MoE as the running example: many total parameters but only a subset active for each generated token ^[src-042].
• Higher sparsity reduces active compute, but increases total parameters and memory capacity requirements ^[src-042].
• Expert parallelism maps different experts to different GPUs, making the traffic pattern all-to-all across the scale-up domain ^[src-042].
• A single rack with full all-to-all connectivity is a natural fit for MoE serving; crossing rack boundaries introduces slower scale-out links ^[src-042].
• Smaller experts reduce the usefulness of tensor parallelism, making expert parallelism and limited pipeline parallelism the main serving strategies ^[src-042].
• Qwen3.8-Max provides a concrete scale example: the article reports 2.4T total parameters but about 95B active per token. Sparse routing lowers active compute, while the full weight set still creates roughly 4.8 TB at 16-bit, 2.4 TB at 8-bit, or 1.2 TB at 4-bit before runtime overhead ^[src-412].

Related entities

• Nvidia Blackwell Nvl72
• Qwen3.8-Max

Related concepts

• LLM Parallelism Strategies
• Scale Up Vs Scale Out Networking
• LLM Inference Economics
• Roofline Analysis For LLM Serving

Source references

• ^[src-042] Dwarkesh Patel — "How GPT, Claude, and Gemini are actually trained and served – Reiner Pope" (2026-04-29)
• ^[src-412] LoKan Sardari — "Qwen3.8-Max face à Claude et GPT-5.6" (2026-08-05)