The specific engineering that turns "naive inference at 5% GPU utilization costing $50 per million tokens" into "vLLM-served 70B model at 60% utilization with 200ms TTFT costing $2 per million tokens." Three primary optimizations: PagedAttention (vLLM — KV cache in fixed-size blocks like OS virtual memory pages, near-zero fragmentation, 2-4× throughput), Continuous batching (Orca/vLLM — iteration-level scheduling, requests join/leave batch dynamically), and Speculative decoding + Quantization (small draft model verifies in parallel for 2-3× latency; INT8/INT4/FP8 for 2-4× memory + throughput). Plus prefill vs decode phase separation, prefill/decode disaggregation (DistServe/Mooncake), and framework composition (vLLM, SGLang, TensorRT-LLM, TGI). Understanding these mechanisms — and how they compose for chat, RAG, and long-form workloads — is Expert-tier competence for modern LLM serving infrastructure.
Inference serving is not "training in reverse" — it\'s a specific engineering problem determined by three tight constraints that differ fundamentally from training: latency (users expect sub-second time-to-first-token, particularly for interactive workloads), throughput (batching many concurrent requests to keep GPUs busy), and memory (KV cache grows with sequence length × batch size × layers × attention heads and can dominate GPU memory at long context). Consider concretely what serving a modern foundation model requires. During training (M.64), a large batch flows through the model, gradients are accumulated, optimizer updates state — the workload is uniform, predictable, and throughput-optimized. During inference, requests arrive independently with variable prompt lengths (100 tokens to 100K+), variable output lengths (10 tokens to 10K+), and independent completion times. Autoregressive decoding is sequential per request: each output token depends on all previous tokens (both prompt and prior generations), read from the KV cache. The naive approach — process each request individually, allocate contiguous KV cache buffer, wait for each request to complete before scheduling the next — achieves 5-10% GPU utilization while costing $50+ per million tokens. Modern serving turns this around through three primary optimizations: (a) PagedAttention (vLLM) manages KV cache as fixed-size blocks like OS virtual memory pages, eliminating fragmentation and enabling non-contiguous storage — 2-4× throughput improvement; (b) Continuous batching (Orca/vLLM) schedules per iteration rather than per request, allowing requests to join and leave the batch dynamically as they arrive and complete — high GPU utilization despite variable request lengths; (c) Speculative decoding + Quantization reduce latency and cost — small draft model proposes N tokens verified in parallel by large model (2-3× latency); INT8/INT4/FP8 quantization reduces memory footprint and improves memory-bound decode throughput (2-4× savings). Understanding these mechanisms — and specifically their composition for different workload profiles — is Expert-tier competence.
The specific engineering task M.65 addresses is understanding how to compose PagedAttention + continuous batching + speculative decoding + quantization for different workload profiles. Modern inference has three primary optimization primitives + phase-aware overlays: (a) PagedAttention (vLLM, Kwon et al. SOSP 2023) — the KV cache management primitive. Allocate cache as fixed-size blocks (typically 16 tokens per block); block table per request maps logical positions to physical blocks; blocks are shared across requests when possible (prefix caching for shared prompt prefixes). Near-zero internal fragmentation vs 60-80% waste for contiguous allocation. Enables 2-4× more concurrent requests and correspondingly higher throughput. Foundational modern primitive. (b) Continuous batching (Orca, Yu et al. OSDI 2022; popularized by vLLM) — the throughput primitive. Traditional "static batching" waits for all requests in the batch to complete before scheduling next batch — GPU sits idle when short requests finish before long ones (30-70% wasted time). Continuous batching (also called "iteration-level scheduling") schedules per iteration: after each forward pass, completed requests leave the batch and new requests join. Requests of variable lengths coexist; GPU utilization stays high. Standard modern discipline. (c) Speculative decoding + Quantization — the latency and cost primitives. Speculative decoding: small draft model (e.g., 1B params) proposes N tokens (typically N=4-8); large model verifies all N tokens in a single forward pass; accept the longest matching prefix. Since decode is memory-bound (weights read once per step), verifying multiple candidate tokens in parallel takes similar time to one — effective 2-3× latency reduction if draft accuracy is decent. Quantization: reduce weight precision from FP16 to INT8 (2× memory), INT4 (4× memory via GPTQ/AWQ), or FP8 (H100 Transformer Engine). 2-4× throughput improvement, minimal quality loss with modern techniques. (d) Prefill vs decode phase separation: prefill is compute-bound; decode is memory-bound. Different optimization strategies for each. Modern schedulers separate them: chunked prefill (split long prompts to interleave with decode), prefill/decode disaggregation (DistServe/Mooncake). Standard 2024+ pattern for frontier-scale serving.
Each earlier attempt fails specifically. Naive single-request loses to no batching + fragmented KV; $50/M. Static batching waits for slowest, pads waste; $20/M. Continuous batching alone still fragments KV; $10/M. The Expert pattern: compose PagedAttention + continuous batching + speculative decoding + quantization matched to workload. PagedAttention manages KV cache as 16-token blocks (95%+ memory efficiency, prefix caching enabled). Continuous batching schedules per iteration (variable-length requests coexist, 55-70% GPU util). Speculative decoding + quantization reduce latency + memory. Framework choice (vLLM/SGLang/TensorRT-LLM/TGI) matches workload. Prefill/decode disaggregation at frontier scale. Standard modern discipline; $2-5/M tokens; production economics. §02 covers PagedAttention + KV cache anatomy in depth. §03 covers continuous batching + speculative decoding + quantization + framework landscape.
The historical arc of LLM serving optimization traces specifically how each primitive addressed a specific bottleneck as the industry scaled from GPT-3 API to modern frontier serving. 2020: GPT-3 API era. OpenAI serves GPT-3 (175B) via a small number of massive GPU instances. Batching is basic; costs are high; latency is unpredictable. 2021-2022: FasterTransformer (NVIDIA). Fused kernels for transformer inference; tensor parallelism at inference (different constraints than training — no gradients, prefill vs decode differences); mixed precision INT8. Foundational NVIDIA inference stack. 2022: Orca paper (OSDI). Yu et al. propose "iteration-level scheduling" (later called continuous batching): schedule per model iteration rather than per request; requests can join and leave the batch dynamically. Theoretical basis for modern serving. 2023: vLLM paper (SOSP). Kwon et al. propose PagedAttention: manage KV cache as fixed-size blocks like OS virtual memory pages. Combined with continuous batching, achieves 2-4× throughput improvement over prior state-of-art. Open source release becomes industry-standard within months. Foundational modern serving stack. 2023: Speculative decoding. Leviathan et al. (Google) and Chen et al. (DeepMind) independently propose speculative sampling: use small draft model to propose tokens, large model verifies in parallel. 2-3× latency for memory-bound decode. 2023: Quantization matures. GPTQ (post-training quantization to INT4), AWQ (activation-aware quantization), SmoothQuant (activation smoothing for INT8), llama.cpp\'s 4-bit quantized inference on CPU/edge. 2024: SGLang (Berkeley). Zheng et al. propose RadixAttention: prefix tree-based KV cache sharing across requests; automatic prefix caching for repeated prompt segments. Structured generation (JSON, regex-constrained decoding). 2024: TensorRT-LLM production maturity. NVIDIA\'s kernel-fused production stack; FP8 support on H100 Transformer Engine; strong integration with Triton Inference Server. 2024: MoE serving hot topic. Mixtral-8x7B, DeepSeek-V2 (238B / 21B active) drive interest in efficient MoE inference. Expert parallelism at serving time; all-to-all routing optimizations. 2024-2025: Prefill/decode disaggregation. DistServe (Berkeley) and Mooncake (Moonshot AI) propose separating prefill and decode across different hardware instances. Standard for frontier-scale serving. 2025: Sub-second TTFT for 100B+ models standard. Modern production serving of Claude, GPT-4, Gemini, DeepSeek — sub-second TTFT for chat, throughput of thousands of tokens/sec/GPU for batch, cost $1-3/M tokens. Long-context serving (100K+ tokens) via chunked prefill + KV cache offloading to CPU memory. Standard modern production. The historical arc explains why modern inference is a composed stack — each primitive addressed a specific bottleneck.
The KV cache is the specific data structure that makes autoregressive generation fast — and specifically the bottleneck that dominates GPU memory during serving. Consider concretely what the KV cache stores. For each token in a sequence, and for each attention layer, the model computes K (key) and V (value) tensors used by the attention mechanism. During decoding, generating token N requires attending to all previous tokens 0..N-1 — the K and V for those tokens must be reread from memory. Rather than recomputing them, they\'re cached. The size formula: 2 × num_layers × num_heads × head_dim × seq_len × batch × precision_bytes. For LLaMA-70B (80 layers, 64 heads, 128 head_dim, FP16): each token consumes 2 × 80 × 64 × 128 × 2 = 2.6MB per token. Sequence of 2048 tokens = ~5GB KV cache per request. Sequence of 32768 tokens = ~85GB — exceeds a single H100 for a single request. This is why KV cache management dominates serving system design. The naive contiguous allocation problem: when a request arrives, the system doesn\'t know its final sequence length. Options: (a) allocate for max_seq_len (typical 4K-32K): most requests only use 500-2K tokens; 60-80% memory waste (internal fragmentation); (b) allocate small buffer, reallocate as needed: memory churn, contiguous requirement can\'t be satisfied when memory is fragmented; allocation fails. Both fail at scale. The PagedAttention solution (vLLM, Kwon et al. SOSP 2023): manage KV cache like OS virtual memory. Allocate cache in fixed-size blocks (typical 16 tokens per block). Each request has a block table mapping logical positions (token index) to physical blocks (arbitrary memory locations). Growing a sequence allocates a new block from the free pool — no contiguous requirement, no reallocation. Blocks can be shared across requests (prefix caching: two requests with the same prompt prefix share the same physical blocks). Near-zero fragmentation. Memory efficiency 95%+ vs 40-50% naive. Enables 4-8× more concurrent requests. Foundational modern primitive.
2 × layers × heads × head_dim × seq_len × batch × precision. Grows linearly with tokens. For LLaMA-70B: ~2.6MB per token. 32K sequence = ~85GB. Dominates GPU memory at long context.
Contiguous allocation for max_seq_len wastes 60-80% memory. Limits concurrent requests to ~10-30 vs 100-500 possible. Standard pre-2023 failure.
Fixed-size blocks (16 tokens typical). Block table per request maps logical→physical. Near-zero fragmentation. 95%+ memory efficiency. Foundational vLLM contribution (Kwon et al. SOSP 2023).
Shared prompt prefixes → shared physical blocks. Skip prefill for cached portion. TTFT drops from seconds to milliseconds. 30-80% hit rate typical. Standard vLLM + SGLang optimization.
Prefix tree data structure for automatic prefix sharing detection. Extends vLLM foundations. Popular for RAG + agent workloads with structured prompts. Zheng et al. 2024.
For long context: overflow least-recently-used blocks to CPU memory. Reload on cache hit. Enables 100K+ token contexts on limited HBM. Standard 2024+ pattern.
The KV cache memory math (mech items i-ii) is worth walking through explicitly because it dictates the concrete constraints of every serving deployment. Consider LLaMA-70B on H100 (80GB HBM). Model weights in FP16: 70B × 2 bytes = 140GB — doesn\'t fit on one GPU, requires TP=8 within node (17.5GB weights per GPU). Available for KV cache after weights + activations: ~40GB per GPU × 8 GPUs = 320GB across the TP=8 group. KV cache per token: 2 × 80 × 64 × 128 × 2 = 2.6MB (FP16). Across TP=8: ~325KB per token per GPU. 320GB total ÷ 2.6MB per token = 123K token capacity across all concurrent requests. For 4K-token sequences: 30 concurrent requests naive; with PagedAttention (95% efficient): ~120 concurrent. For 32K-token sequences: 3 concurrent naive; with PagedAttention: ~12 concurrent. Long context is expensive — this is why 100K+ context requires KV cache offloading (mech item vi). Also why FP8/INT4 quantization of KV cache (2-4× reduction) matters significantly for long-context serving. The prefix caching math (mech item iv): for chat workloads with common 500-token system prompt used across N concurrent requests: without prefix caching, N × 500 × 2.6MB = 1.3N GB of KV cache dedicated to duplicated prefix (waste). With prefix caching: 500 × 2.6MB = 1.3GB shared once across all requests (98% savings on prefix portion). For RAG workload where 100 requests each query the same 8K-token document context: without caching, 100 × 8K × 2.6MB = 2.1TB (impossible); with caching, 8K × 2.6MB = 21GB shared (feasible). Prefix caching literally makes long-context RAG serving feasible. Standard modern optimization. Understanding these numbers — memory footprint per token, effect of quantization, effect of prefix caching — is Expert-tier competence.
The PagedAttention block management (mech item iii) deserves specific attention because it\'s the specific data structure that makes modern serving throughput possible. Consider the concrete mechanics: (a) Block size choice: fixed at compile time (typical 16 tokens). Small blocks → less waste in last partial block but more block table overhead. Large blocks → more waste but simpler. 16 tokens is the vLLM default sweet spot. (b) Block table structure: per-request array. For request with 500 tokens: 32 block IDs stored (each ID ~2 bytes → 64 bytes per request block table). Negligible overhead. (c) Allocation: when a request needs a new block (every 16 generated tokens), allocate from free pool. O(1) via free list. When request completes: return all blocks to pool. (d) Attention computation: modified attention kernel (custom CUDA) reads K/V from non-contiguous blocks via block table indirection. Adds small overhead (~5% vs contiguous) but enables massive concurrency gains. vLLM\'s custom PagedAttention kernel is the specific engineering contribution. (e) Prefix sharing: reference-counted blocks. When two requests share prefix blocks: refcount = 2. When one completes: refcount decrements. When refcount reaches 0: block returned to free pool. (f) Preemption: when memory is full and a new request arrives, vLLM can preempt (swap out to CPU or drop) least-important requests. Standard priority-based scheduling. (g) Copy-on-write: for parallel sampling (n>1 generations from same prompt): initially share prefix blocks; when generations diverge, copy the affected block. Extends prefix caching to multi-sample scenarios. (h) The Expert insight: PagedAttention is not "just a memory allocator" — it\'s a foundational primitive that enables (i) high concurrency (100-1000+ requests), (ii) prefix caching (30-80% cache hit rates), (iii) preemption (SLA-aware scheduling), (iv) parallel sampling efficiency. Together these enable modern serving economics ($2/M tokens vs $20-50/M naive). Understanding this mechanism is Expert-tier competence.
Beyond PagedAttention, three additional primitives specifically address the throughput, latency, and cost constraints of modern LLM serving: continuous batching (throughput), speculative decoding (latency), and quantization (memory + throughput + cost). Each has specific mechanics + specific fit; their composition is what modern frameworks (vLLM, SGLang, TensorRT-LLM, TGI) provide. (a) Continuous batching (Orca / vLLM): schedule per iteration rather than per request. Traditional "static batching" waits for all N requests to complete their generations before running the next batch of N — 30-70% of time wasted when requests have variable lengths. Continuous batching runs one forward pass over the current active batch; after each pass, requests that completed leave the batch (their KV cache blocks freed), new requests join. Handles the fundamental variability of request lengths gracefully. Prefill of a new request can interleave with decode of ongoing requests. Enables 50-70% GPU utilization for variable workloads vs 20-30% static. Foundational. (b) Speculative decoding (Leviathan et al. Google 2023; Chen et al. DeepMind 2023): leverage the fact that decode is memory-bound. Small "draft" model (e.g., LLaMA-1B for LLaMA-70B target) autoregressively generates N candidate tokens (typically N=4-8). Large "target" model processes those N tokens in a single forward pass (verifying their K/V + computing what the target model would have generated). Accept the longest matching prefix; reject the rest. Since decode is memory-bandwidth-bound (weights read once per token), verifying multiple candidates in parallel has similar cost to one — effective 2-3× latency reduction when acceptance rate is decent (60-80% typical). Standard for latency-critical workloads. (c) Quantization: reduce precision of weights and/or KV cache and/or activations. FP16→FP8 (H100 Transformer Engine): 2× memory, 1.5-2× throughput, ~1% quality loss. FP16→INT8 (SmoothQuant, LLM.int8()): 2× memory, 1.5-2× throughput, ~1-2% quality loss. FP16→INT4 (GPTQ, AWQ): 4× memory, 2-3× throughput, ~3-5% quality loss depending on method. Since decode is memory-bound: throughput scales roughly with 1/precision. Also apply to KV cache: FP16→FP8 KV cache halves memory footprint, enabling 2× longer contexts or 2× more concurrent requests.
Iteration-level scheduling. Requests join/leave batch dynamically. Prefill interleaves with decode. GPU util 50-70% vs 20-30% static. 3-5× throughput. Orca 2022 → vLLM standard.
Small draft model proposes N tokens; large target verifies in parallel. Decode memory-bound → verify cost ≈ single token. 60-80% acceptance → 2-3× latency. Standard for chat.
FP8 (H100 TE): 2× · minimal loss. INT8: 2× · ~1% loss. INT4 (AWQ/GPTQ): 4× · ~3-5% loss. Decode is memory-bound → throughput ∝ 1/precision. Standard modern discipline.
PagedAttention foundational. Continuous batching. Prefix caching. Most popular open-source serving. GitHub: vllm-project/vllm. Standard 2023+ default choice.
RadixAttention prefix caching. Structured generation (JSON, regex). Excellent for RAG + agent workloads. GitHub: sgl-project/sglang. Strong 2024 alternative to vLLM.
Kernel-fused production stack. FP8 on H100 TE. Strong Triton Inference Server integration. Best NVIDIA-native perf. Standard for NVIDIA-native production. Closed source, NVIDIA-only.
The prefill vs decode phase separation (foundational architectural insight) is worth specific attention because it dictates every modern serving system\'s scheduler design. Consider the specific differences: (a) Prefill computationally: process entire N-token prompt at once. Matrix operations are (batch × N × hidden) × (hidden × hidden) = O(N × hidden²) per layer. Attention specifically is O(N² × hidden) — quadratic in prompt length. Compute-bound: matmul units fully utilized. TTFT (Time-to-First-Token) is dominated by prefill cost. Grows quadratically with prompt length. Long-context prefill (32K+ tokens) can take seconds. (b) Decode computationally: generate one token at a time. Matmul is (batch × 1 × hidden) × (hidden × hidden) — tiny per step. Attention is O(N × hidden) per step (attend to N cached tokens). Memory-bandwidth-bound: for each token, read full model weights + full KV cache from HBM into compute; compute takes microseconds but memory read dominates. Weights: 140GB FP16 for 70B; HBM bandwidth 3.35 TB/s → ~40ms to read weights per token. GPU compute severely underutilized (1-10% of peak matmul FLOPs). (c) Why batching helps decode dramatically: memory bandwidth is amortized across the batch. Reading 140GB of weights once serves the whole batch\'s decode step. Batch of 100 → same 40ms, 100 tokens produced → 2500 tokens/sec. Continuous batching + PagedAttention enable large batches. (d) Why prefill doesn\'t benefit as much from batching: prefill is compute-bound already; batching increases compute proportionally without free lunch. Also prefill dominates latency for interactive workloads — batch prefills of long prompts add TTFT for waiting requests. (e) Chunked prefill: split long prompts into chunks (say 512 tokens each); process chunks incrementally interleaved with decode steps for other requests. Amortizes long-prefill latency across many decode steps for others. Standard modern optimization. (f) Prefill/decode disaggregation (DistServe/Mooncake): at frontier scale, prefill and decode have different SLA targets (TTFT vs TPOT) and different hardware needs (compute-rich vs memory-rich). Solution: separate them across different hardware pools. Prefill instances: process incoming requests, produce KV cache. Transfer KV cache to decode instance. Decode instance: continuous decode of many requests. Adds inter-instance communication cost but enables independent scaling of each phase, matches SLAs. Standard for 100B+ MoE serving. Understanding this — that prefill and decode are fundamentally different problems requiring different optimization strategies — is Expert-tier competence.
The framework choice heuristic (mech items iv-vi) deserves specific attention because it\'s where production engineering decisions actually get made. Consider the specific tradeoffs: (a) vLLM: canonical modern serving framework. PagedAttention foundational contribution. Continuous batching. Prefix caching. Broad model support via HuggingFace integration. Popular for prototyping + production. GitHub 30K+ stars. Standard 2023+ default choice for open-source LLM serving. Weakness: RadixAttention (SGLang) does better prefix caching for RAG; TensorRT-LLM has better FP8 kernels on H100 for pure NVIDIA. (b) SGLang: newer Berkeley project. RadixAttention: prefix tree data structure for automatic prefix sharing detection (extends vLLM foundations). Structured generation: constrained decoding for JSON schemas, regex matches, tool use — critical for agent + RAG workloads. Zheng et al. 2024. Excellent for prompt-heavy workloads with high prefix overlap. Strong 2024 alternative to vLLM for specific workloads. (c) TensorRT-LLM: NVIDIA production stack. Kernel-fused (fused attention + matmul + activation). FP8 on H100 Transformer Engine. Best raw performance on NVIDIA hardware. Strong Triton Inference Server integration for production deployment. Closed source, NVIDIA-only. Standard for NVIDIA-native production requiring maximum throughput. (d) TGI (HuggingFace Text Generation Inference): HuggingFace\'s inference server. Easy deployment via HF ecosystem. Good compatibility with HF models. Less optimized than vLLM/TRT-LLM at peak but simpler ops. Standard for teams already in HF ecosystem. (e) DeepSpeed-Inference: Microsoft\'s inference. Strong MoE support. Composes with DeepSpeed training stack. Less popular than vLLM in 2024-2025 for new deployments but good for MoE-specific workloads. (f) llama.cpp: CPU + edge inference. GGUF format. 4-bit quantization. Standard for on-device serving. (g) Framework choice heuristic: default vLLM for general open-source serving; SGLang for RAG + agent workloads with structured output requirements; TensorRT-LLM for maximum throughput on NVIDIA-native production; TGI for HF ecosystem alignment; llama.cpp for edge. Choose based on workload + hardware + team + ecosystem alignment. Standard modern discipline.
Below: each of three inference optimization primitives (PagedAttention · Continuous batching · Speculative decoding + Quantization) evaluated against three workload profiles (7B chat @ high QPS · 70B RAG @ long context · 400B MoE @ mixed frontier). Watch how each primitive fits each workload — Continuous batching is IDEAL for chat (high concurrency canonical), PagedAttention is KEY for RAG (long-context KV memory dominates + prefix caching wins), Speculative + Quantization is IDEAL for MoE frontier (latency + cost critical at scale). Off-diagonals show partial fit — each primitive still helps, but with less leverage than its ideal workload. The takeaway: match optimization stack to workload characteristics; compose all three at production scale.
The failure modes of LLM serving are specific mechanisms by which "we\'re serving Llama-70B on 8 H100s" turns into "we\'re at 15% GPU util paying $30 per million tokens." Each anti-pattern is a real production pattern; Expert engineers avoid them by using PagedAttention for KV management, continuous batching for throughput, speculative decoding for latency, quantization for cost, and treating prefill/decode as fundamentally different phases. Recognizing these saves months of "why is our serving so expensive" debugging.
Static batching (fixed-batch, wait-for-slowest) wastes 30-70% of GPU time when requests have variable lengths — which is always the case for real LLM serving. The specific fix is continuous batching (iteration-level scheduling): after each forward pass, completed requests leave the batch and new requests join. Specifically: (a) THE FUNDAMENTAL PROBLEM. Static batching groups N requests together at start; runs forward passes over batch; when all N complete, schedules next batch. But request lengths vary drastically — median chat completion 100-500 tokens, tail can be 5000-10000. When 7 requests complete at 100 tokens and 1 continues to 5000: those 7 GPUs sit idle for 4900 iterations. Total wasted GPU time: (5000-100) × 7 / (5000 × 8) = 86% of the slow request\'s time, 60% of total batch time. (b) THE CONTINUOUS BATCHING FIX. Iteration-level scheduling (Orca, vLLM): after each forward pass, examine which requests completed (hit stop token, reached max length). Freed slots immediately accept new requests from the queue. Requests of different generation lengths coexist naturally. The batch composition changes every iteration. GPU utilization jumps from 20-30% to 50-70%. Throughput 3-5× higher. Standard modern discipline. (c) THE PREFILL INTERLEAVING. New request arrives while decode is ongoing: modern schedulers (vLLM v0.5+, SGLang) interleave prefill of new request with decode of existing requests. Prefill is compute-bound, decode is memory-bound → they use complementary resources. Chunked prefill: split long prompts into smaller chunks to interleave more finely. Standard modern optimization. (d) THE VLLM API. from vllm import LLM, SamplingParams; llm = LLM(model="meta-llama/Llama-3-70b", tensor_parallel_size=8); outputs = llm.generate(prompts, sampling_params). Continuous batching enabled by default; server mode (vllm serve) handles concurrent HTTP requests. Standard modern pattern. (e) THE MEASUREMENT. Track GPU utilization (nvidia-smi during load); target 50-70% for well-tuned modern serving. Track requests-per-second, tokens-per-second, TTFT, TPOT. Alert on util drops below 40%. Standard modern discipline. Understanding this fix — that static batching is objectively wrong for variable-length workloads and continuous batching is the specific answer — is Expert-tier competence. Anti-pattern §05.i captures the failure to use iteration-level scheduling.
Contiguous KV cache allocation wastes 60-80% of GPU memory on internal fragmentation — allocating for max_seq_len per request when most requests only use a fraction. The specific fix is PagedAttention: manage KV cache in fixed-size blocks (16 tokens) like OS virtual memory pages, near-zero fragmentation, 4-8× more concurrent requests. Specifically: (a) THE ROOT CAUSE. When request arrives, system doesn\'t know final sequence length. Naive allocation: contiguous buffer for max_seq_len (typical 4K-32K). Request only uses 500-2K tokens → 60-80% of the buffer wasted. For LLaMA-70B on 8× H100 (320GB KV budget), 32K max sequence = 30 requests × 32K × 2.6MB/token = 2.5TB attempted vs 320GB available. Only 30 concurrent effectively; 60-80% of those buffers wasted. (b) THE PAGEDATTENTION FIX. Divide KV cache into fixed-size blocks (16 tokens per block). Physical block pool: pre-allocated free list of blocks. Per-request block table: maps logical token positions to physical block IDs. Growing sequence allocates blocks from pool on demand. When request completes: blocks returned to pool. Near-zero fragmentation (only ~50% average waste in last partial block per request, ~3% at scale). Memory efficiency 95%+. Enables 4-8× more concurrent requests. Standard vLLM contribution (Kwon et al. SOSP 2023). (c) THE PREFIX CACHING BONUS. Shared prompt prefixes (system prompts, RAG contexts, conversation history) → shared physical blocks. Two requests with same first 500 tokens: block table for both points to same physical blocks B100-B131. Blocks physically stored once. TTFT for cached prefix: milliseconds (skip prefill entirely). Standard vLLM prefix_caching_enabled=True. 30-80% cache hit rate typical in RAG workloads. Enormous compute + memory savings. (d) THE VLLM CONFIGURATION. llm = LLM(model="meta-llama/Llama-3-70b", tensor_parallel_size=8, enable_prefix_caching=True, block_size=16). Block size 16 is the default sweet spot. Prefix caching auto-detects shared prefixes across requests. Standard modern config. (e) THE SGLANG ALTERNATIVE. SGLang\'s RadixAttention uses radix tree structure for automatic prefix sharing detection — more sophisticated than vLLM\'s hash-based caching. Excellent for RAG + agent workloads with highly structured prompts. (f) THE MEASUREMENT. Track KV cache memory efficiency (used blocks / total blocks × block_size × ideal_usage). Target 90%+. Track prefix cache hit rate. Track concurrent request count. Standard modern discipline. Understanding this fix — that PagedAttention is the specific answer to "why can\'t I serve more concurrent requests" — is Expert-tier competence. Anti-pattern §05.ii captures the failure to use paged KV management.
Decode phase is memory-bandwidth-bound — GPU compute severely underutilized while HBM reads dominate. Speculative decoding exploits this by having a small draft model propose N tokens, then having the large target model verify all N in a single forward pass. Since decode is memory-bound, verifying N tokens costs approximately the same as verifying 1 → effective 2-3× latency reduction. Specifically: (a) THE MECHANISM. Small draft model (e.g., LLaMA-1B for LLaMA-70B target — 70× smaller, ~70× faster to run standalone) autoregressively generates N candidate tokens (typically N=4-8) very quickly. Large target model then processes all N candidate tokens in a single forward pass — verifying each token\'s K/V and computing what target would have generated. Accept longest prefix where candidates match target\'s greedy (or sampled) output. Reject rest, continue from accepted position. (b) THE KEY INSIGHT. Target model decode is memory-bandwidth-bound: reading 140GB of weights from HBM per token takes ~40ms; compute of matmul on 1 token vs N tokens takes microseconds either way. So verifying N tokens in one forward pass costs ~40ms (same as verifying 1). If acceptance rate is 75%: ~6 tokens accepted per verify → 6 tokens per 40ms = 150 tokens/sec vs 25 tokens/sec baseline. Practical speedup 2-3× after overhead (draft model time, rejection handling). (c) THE ACCEPTANCE RATE. Depends on draft-target model similarity. For LLaMA-1B / LLaMA-70B: typically 60-80% acceptance. Higher for constrained-output tasks (code, structured), lower for creative writing. Extensions: EAGLE (feature-level auto-regression from target hidden states, higher acceptance ~85%), Medusa (multi-head decoding on target model — no separate draft, learned decoding heads, ~2× speedup), lookahead decoding (Jacobi iteration, no draft model needed but lower speedup ~1.5×). (d) THE VLLM CONFIGURATION. llm = LLM(model="meta-llama/Llama-3-70b", tensor_parallel_size=8, speculative_model="meta-llama/Llama-3-1b", num_speculative_tokens=4). Standard modern config. TensorRT-LLM supports Medusa + EAGLE via built-in kernels. (e) THE FIT HEURISTIC. Speculative decoding: essential for latency-critical interactive workloads (chat interface, code completion). Marginal for batch workloads (throughput-focused — spec adds compute cost per token, batched decode already amortizes memory reads). Not helpful when decode is compute-bound (very short sequences with tiny batch sizes). Standard modern discipline. (f) THE MEASUREMENT. Track TPOT with and without spec; track acceptance rate; alert on rate drops (indicates draft-target divergence). Standard modern discipline. Understanding this — that speculative decoding is specifically effective because decode is memory-bound — is Expert-tier competence. Anti-pattern §05.iii captures the failure to use spec for latency-critical serving.
Ignoring quantization leaves 2-4× throughput on the table. Modern H100 serving typically uses FP8 (via Transformer Engine) for 2× throughput at ~1% quality loss; INT4 (via AWQ or GPTQ) for 4× throughput at ~3-5% quality loss. Since decode is memory-bandwidth-bound, throughput scales approximately with 1/precision — the specific mechanism that reduces cost per token. Specifically: (a) THE MEMORY-BANDWIDTH MATH. Decode phase reads full model weights from HBM per token per request. For LLaMA-70B FP16: 140GB weights / 3.35 TB/s HBM = 42ms per token per batch (weights amortized across batch). For FP8: 70GB weights / 3.35 TB/s = 21ms → 2× faster. For INT4: 35GB weights / 3.35 TB/s = 10.5ms → 4× faster. Throughput scales directly with 1/precision because decode is memory-bound. (b) THE FP8 ON H100. H100 Transformer Engine provides native FP8 matmul with automatic precision management (FP16 accumulators, FP8 weights/activations). Configurations: E4M3 for forward (4-bit exponent, 3-bit mantissa — better range), E5M2 for backward gradients. Quality loss ~1% for most models (calibrated). Standard 2024-2025 serving discipline for H100 deployments. vLLM: llm = LLM(model="...", quantization="fp8"). TensorRT-LLM has best FP8 kernel implementation. (c) THE INT4 VIA AWQ/GPTQ. Post-training quantization. GPTQ (Frantar et al. 2022): layer-wise Hessian-based error minimization; 4-bit weights. AWQ (Lin et al. 2023): activation-aware — protects weights corresponding to salient activation channels; typically ~1% quality improvement over GPTQ at 4-bit. Both maintain FP16 activations (weight-only quantization). Quality loss: 2-5% for well-calibrated models, up to 10% for aggressive settings. vLLM: llm = LLM(model="TheBloke/Llama-3-70B-AWQ", quantization="awq"). Standard for memory-constrained deployments. (d) THE KV CACHE QUANTIZATION. Beyond weights: quantize KV cache. FP8 KV cache halves memory footprint → 2× longer contexts or 2× more concurrent requests. INT8 KV cache: similar 2× savings, slight quality loss. Standard 2024+ pattern. vLLM: kv_cache_dtype="fp8". (e) THE COMPOSITION. Combine weight quantization (FP8/INT4) + KV cache quantization (FP8) + PagedAttention + continuous batching + speculative decoding for full modern stack. Cost drops from $10-15/M tokens (FP16 baseline) to $2-5/M tokens (fully optimized). (f) THE QUALITY-COST TRADEOFF. Task-dependent: (i) chat / general QA: FP8 losses undetectable; INT4 losses marginal; (ii) code generation: FP8 fine; INT4 shows some errors on complex code; (iii) math/reasoning: FP8 fine; INT4 shows accuracy drops; (iv) creative writing: subjective, hard to measure. Standard evaluation: benchmarks (MMLU, HumanEval, GSM8K) with quantized vs FP16 baseline. Choose precision based on quality budget + cost target. Standard modern discipline. Understanding this — that quantization is the specific lever for reducing serving cost, and that the tradeoff is quality vs cost — is Expert-tier competence. Anti-pattern §05.iv captures the failure to leverage quantization.
Prefill and decode have fundamentally different characteristics: prefill is compute-bound with quadratic latency in prompt length; decode is memory-bandwidth-bound with linear per-token latency. At frontier scale (100B+ MoE), separating them across different hardware pools (prefill/decode disaggregation) enables independent SLA management and specialized hardware selection. Standard 2024+ pattern for frontier serving. Specifically: (a) THE MIXED-PHASE PROBLEM. Long-context prefill (32K tokens on 70B model) takes 2-3 seconds on H100. During that time, GPU is fully occupied — other requests\' decode stalls. Continuous batching helps for typical workloads but breaks down when prefill dominates. TTFT SLA for waiting requests violated. Standard failure of mixed-phase serving at scale. (b) THE DISAGGREGATION PATTERN. Separate physical machines: (i) PREFILL POOL — compute-rich H100s, optimized for burst matmul work; process incoming requests, produce KV cache; (ii) DECODE POOL — memory-rich H100s (larger HBM per GPU, higher HBM bandwidth), optimized for continuous decode. Between them: transfer KV cache via network. Router: sends new request to prefill pool, receives KV cache, forwards to decode pool. (c) THE DISTSERVE (Berkeley 2024) + MOONCAKE (Moonshot AI 2024) IMPLEMENTATIONS. DistServe: proves disaggregation improves both TTFT and TPOT at scale — prefill pool sizes independently of decode pool. Mooncake: production Chinese-language LLM (Moonshot) serving; uses RDMA for fast KV cache transfer between prefill and decode nodes; heavy KV cache offloading + prefix caching. Standard 2024-2025 frontier patterns. (d) THE KV TRANSFER COST. Between prefill and decode nodes: transfer full KV cache for request. For LLaMA-70B, 8K prompt: 8000 × 2.6MB = 20GB. At 400 Gb/s InfiniBand (50 GB/s bulk): 400ms transfer time. Significant overhead. Mitigations: (i) use PCIe P2P + RDMA for GPU-to-GPU direct transfer (100 GB/s+); (ii) compress KV cache in transit (INT8/FP8 quantization); (iii) overlap transfer with decode of first tokens; (iv) prefix cache hits avoid transfer entirely (shared prefix served from cache). Standard modern discipline. (e) THE SCHEDULER COMPLEXITY. Prefill/decode disaggregation requires more sophisticated scheduling: (i) prefill queue with priority (long prompts scheduled to avoid HOL blocking); (ii) chunked prefill (split long prompts, interleave with decode); (iii) prefill-decode ratio autoscaling (match load characteristics); (iv) failure recovery (KV cache transfer + decode node crash handling). Standard modern engineering. (f) THE FIT. Disaggregation: essential at 100B+ MoE scale with mixed prompt lengths; overkill for single-model chat serving where prompts are consistent. Standard 2024+ pattern for frontier labs (Anthropic, OpenAI, Google, DeepSeek, Moonshot). vLLM has experimental disaggregation support in 2024+; SGLang provides similar. Standard modern discipline. (g) THE MEASUREMENT. Track TTFT p50/p99 separately from TPOT p50/p99; track KV cache transfer latency; track prefill/decode pool utilization independently. Standard modern discipline. Understanding this — that prefill/decode disaggregation is the specific pattern for frontier-scale serving with mixed workloads — is Expert-tier competence. Anti-pattern §05.v captures the failure to disaggregate at scale.
The composite pattern across all five is that serving failure modes reflect specific engineering gaps in scheduling (static vs continuous), memory management (contiguous vs paged), latency optimization (missing spec decoding), cost optimization (ignored quantization), and phase separation (mixed prefill/decode). Static batching wastes GPU time on slow requests. Contiguous KV wastes memory on fragmentation. No speculative decoding leaves latency unfixed. FP16 baseline serving costs 4× more than optimized. Not disaggregating hurts SLAs at scale. Each has specific fixes: (a) continuous batching (Orca/vLLM iteration-level); (b) PagedAttention (16-token blocks + prefix caching); (c) speculative decoding (small draft + parallel verify); (d) FP8/INT4 quantization (weights + KV); (e) prefill/decode disaggregation at frontier scale. Getting LLM serving right is the specific engineering discipline that turns "our 70B serving is expensive and slow" into "we\'re serving Llama-70B on 8 H100s at 60% GPU utilization with 200ms TTFT, 40 tokens/sec TPOT, and $2/M tokens — competitive production economics."
The terms that show up in every LLM serving design review, every vLLM config, every latency SLA discussion.
2 × layers × heads × head_dim × seq × batch × precision. For LLaMA-70B: ~2.6MB per token FP16. Dominates GPU memory at long context.Test the LLM serving understanding. Click an answer; explanation drops in instantly.
Perfect. PagedAttention, continuous batching, speculative decoding, quantization, prefill/decode disaggregation — the specific engineering for modern LLM serving infrastructure. Next: M.66.
The composite understanding that turns "naive serving at 5% GPU utilization costing $50 per million tokens" into "vLLM-served 70B model at 60% GPU utilization with 200ms TTFT and $2 per million tokens — competitive production economics."
Prefill is compute-bound (O(N²) attention over prompt); TTFT dominant. Decode is memory-bandwidth-bound (KV cache reads per token); TPOT dominant. Different optimization strategies for each. Chunked prefill interleaves with decode; disaggregation separates them at frontier scale. Standard modern discipline.
PagedAttention manages KV cache as 16-token blocks (near-zero fragmentation, 95%+ efficiency, prefix caching). Continuous batching schedules per iteration (requests join/leave dynamically, 50-70% GPU util). Together: 4-8× more concurrent requests, 3-5× throughput vs naive. vLLM standard. Standard modern discipline.
Speculative decoding: small draft proposes N tokens, target verifies in parallel; 2-3× latency reduction. Quantization: FP8 (H100 TE), INT4 (AWQ/GPTQ) reduces memory-bound decode cost; 2-4× throughput. Compose with PagedAttention + continuous batching for full modern stack. Standard modern discipline.