1. What inference optimization is (the short version)

• Inference has two stages: "prefill", which reads the whole input at once and is limited by arithmetic, and "decode", which writes the answer one token at a time and is limited by memory reads. Most discussions of speed and cost start from this difference.Sourced[Refs. 1, 13]
• Speed, cost and waiting time cannot all be optimized at once: serving more requests together (a larger batch) improves output and cost per GPU but lengthens the gap between tokens for each user (estimated in Chapter 4).Our calculation
• Memory decides how many users fit at once: the in-progress state of a conversation (the KV cache) grows with every token and occupies GPU memory. How it is stored (for example PagedAttention) and whether the two stages run on separate machines are the heart of practical serving design.Sourced[Refs. 1, 9]

2. The two stages of inference: prefill and decode

A language model writes by predicting the next token, one at a time. Each new token requires a pass through the whole model; Leviathan et al. note that producing K tokens takes K serial runs of the model.Sourced[Ref. 3] The first step, reading the whole input, is different, because all input tokens can be processed in parallel. The Sarathi-Serve paper describes prefill as high-latency but saturating GPU compute, and decode as low-latency but with low compute utilization because each iteration processes only one token per request.Sourced[Ref. 10]

Splitwise describes the same split: prompt computation is compute-intensive and token generation is memory-intensive, each with distinct latency, throughput, memory and power characteristics.Sourced[Ref. 8] For LLM latency, the MLPerf Inference rules define two metrics: TTFT, the latency of the first token, and TPOT, the average interval between generated tokens.Sourced[Ref. 14] Roughly, TTFT corresponds to how long a user waits before anything appears and TPOT to how fast text streams out (this mapping is our commentary).

3. The KV cache: the in-progress state that occupies memory

A Transformer computes attention between each new token and every earlier token. Recomputing the keys and values of past tokens each time would be wasteful, so they are stored and reused: this is the KV cache. The size per token is 2 (key and value) x layers x number of KV heads x head dimension x bytes per element (our summary).

The vLLM paper shows that for OPT-13B in FP16 one token's KV cache is 800 KB (2 x 5,120 x 40 layers x 2 bytes), so a 2,048-token request can need up to 1.6 GB.Sourced[Ref. 1] It also reports, for a 13B model on a 40 GB NVIDIA A100, that about 65% of memory held the weights and close to 30% held the per-request dynamic state, mainly the KV cache.Sourced[Ref. 1] The larger the KV cache, the fewer users can be held at once and the lower the throughput.

Recent models therefore use designs that shrink the KV cache. Grouped-query attention (GQA) of Ainslie et al. uses fewer key/value heads than query heads, aiming for quality close to multi-head attention at speed comparable to multi-query attention.Sourced[Ref. 6] The Llama 3 paper says it uses GQA with 8 key-value heads to improve inference speed and reduce the size of key-value caches during decoding.Sourced[Ref. 7]

4. Batching and throughput: more at once is cheaper, but each user waits longer

During decode, all the model weights are read from memory for every token. Whether one user or ten are processed together, the weights are read once. Raising the number of requests processed together (the batch) therefore raises the tokens per second per GPU. The vLLM paper also states that high-throughput serving requires batching sufficiently many requests at a time.Sourced[Ref. 1]

With classic "static batching", everyone waits until the longest response in the batch finishes. Orca (OSDI 2022) proposed iteration-level scheduling, which re-schedules after every single model iteration: requests that finish early can be returned at once and new requests join at the next iteration.Sourced[Ref. 2] This is the idea behind the "continuous batching" used by many serving systems today (that mapping is our commentary).

All Our calculation. Llama 3 8B (BF16), H100 SXM (3.35 TB/s bandwidth; dense compute taken as half the with-sparsity figure), every user at 2,048 tokens, one GPU, and an illustrative price of US$3 per GPU-hour. Not measured values.

5. Using memory well: PagedAttention, prefix reuse and FlashAttention

Earlier systems reserved each request's KV cache as one contiguous region sized for the maximum length. Real responses are often much shorter, which wastes reserved-but-unused space (internal fragmentation) and leaves gaps between regions (external fragmentation). In the vLLM paper's measurements, only 20.4% to 38.2% of the KV-cache memory in existing systems held actual token states.Sourced[Ref. 1]

PagedAttention follows operating-system virtual memory and paging: the KV cache is split into fixed-size blocks that can sit in non-contiguous places, linked to their logical order by a block table. Waste is then confined mostly to the last block, and blocks can be shared within and across requests. The vLLM authors report 2x to 4x higher throughput than the state-of-the-art systems FasterTransformer and Orca at the same latency level (values under the authors' own conditions).Sourced[Ref. 1]

View original SVG, values and source annotations

When many requests share the same beginning (a system prompt or a common document), the KV cache for that part can be reused instead of recomputed. The SGLang paper proposes RadixAttention for this and reports up to 6.4x higher throughput on tasks such as multi-turn chat and retrieval-augmented pipelines (under the authors' conditions).Sourced[Ref. 12] Whether reuse pays off depends on how much of the prompt is shared and how much cache memory can be kept (our commentary).

The attention computation itself is also limited by memory traffic. FlashAttention is an IO-aware, exact (not approximate) attention algorithm that uses tiling to reduce reads and writes between the GPU's large HBM and its small, fast on-chip SRAM.Sourced[Ref. 5] Because it reduces memory movement without changing the result, it belongs with "same output, faster" techniques such as speculative decoding in the next chapter (this grouping is our own).

6. Speculative decoding: a small model drafts, the large model checks in one pass

Decode is slow because of weight reads, while arithmetic has headroom (Chapter 4). The idea: a cheap small model (the draft) proposes several tokens ahead, and the large model checks them in parallel, advancing several tokens at once when they are right. Leviathan et al. designed the accept/reject rule so that the output distribution is unchanged and report a 2x to 3x speed-up on T5-XXL,Sourced[Ref. 3] and Chen et al. report 2x to 2.5x on the 70-billion-parameter Chinchilla in a distributed setupSourced[Ref. 4] (both under the authors' conditions).

Leviathan et al. let the acceptance rate α be the average probability that a drafted token is accepted by the large model, c the cost of one draft step relative to one large-model step, and γ the number of tokens drafted per round. They derive the expected tokens per round as (1 - α^(γ+1)) / (1 - α) and the expected wall-time improvement as (1 - α^(γ+1)) / ((1 - α)(γc + 1)), under the simplification that acceptance is independent and equally likely per token.Sourced[Ref. 3]

All Our calculation (the formula of Leviathan et al. [Ref. 3] with c = 0.05). The cost ratio c = 0.05 is illustrative; the paper reports c below 0.05 for drafts about two orders of magnitude smaller.

Three cautions. First, the verification step adds arithmetic, so total computation rises; Leviathan et al. frame the method as trading extra arithmetic for lower latency.Sourced[Ref. 3] It helps when decode is stalled on memory rather than arithmetic, and the benefit can shrink when large batches leave little spare compute (this reading follows from the paper's statement about extra operations and is our commentary, not a measurement). Second, the draft can be a separate small model, or the large model's own internal features can predict the next tokens (for example EAGLE), so implementations differ.Sourced[Ref. 19] Third, in evaluation, the MLPerf Inference rules allow speculative decoding only for specific workloads, with the reference prediction head and configuration fixed by the rules.Sourced[Ref. 14]

7. Disaggregating prefill and decode, and chunked prefill: avoiding interference

Mixing prefill and decode on the same GPU lets a long prefill stall the responses of other users who are mid-decode. The DistServe paper points out that existing systems batch both phases together on the same GPUs, causing strong interference and coupling the resource allocation and parallelism plans of the two phases.Sourced[Ref. 9] There are two broad remedies.

• Chunked prefill: Sarathi-Serve splits a prefill into near-equal chunks and builds "stall-free" schedules that add new requests without pausing ongoing decodes. For Mistral-7B on a single A100 the authors report 2.6x the serving capacity (work handled within tail-latency limits) of vLLM (under the authors' conditions).Sourced[Ref. 10]
• Disaggregation: run prefill and decode on separate GPU groups and pass the KV cache between them. Splitwise reports 1.4x higher throughput at 20% lower cost, or 2.35x more throughput at the same cost and power budget;Sourced[Ref. 8] DistServe reports serving up to 7.4x more requests, or meeting a 12.6x tighter latency target, within latency constraintsSourced[Ref. 9] (both under the authors' conditions).

Mooncake is the serving platform of Moonshot AI's Kimi service. It separates prefill and decode clusters and uses the otherwise idle CPU, DRAM and SSD of the GPU cluster as a place to keep KV cache. The authors state that, under real workloads, the design let Kimi handle 75% more requests (the company's own report).Sourced[Ref. 11]

All Sourced (each paper's authors' reports). Baselines, models, hardware and workloads differ between papers, so the factors in the table cannot be compared with each other to rank the methods.

8. Reading latency, throughput and cost: the conditions that must accompany a number

As Chapters 4 to 7 showed, the same system can give numbers that differ by several times depending on conditions. The table lists what to check when you see a claim about speed or cost (our summary).

This list is our own summary of Chapters 4 to 7. Prices change often, so this article quotes no provider's rates.

9. Limits, and what could not be confirmed

• No real measurements are shown: our calculations are idealized estimates that assume memory-bound decode. Individual MLPerf submission results and cross-vendor hardware comparisons are not shown because their settings are not aligned.
• Techniques not covered in depth: quantization details, expert parallelism for MoE, communication costs of tensor and pipeline parallelism, KV-cache quantization and compression, and sliding-window attention (see our explainer on model efficiency).
• Differences between serving stacks: features and version-by-version differences of vLLM (open source under Apache-2.0; the repository was seen to be updated on 14 October 2026),Sourced[Ref. 20] SGLang and other stacks were not verified.
• Prices: API rates change quickly and need an official price page and a date, so none are given. The cost example uses an illustrative US$3 per GPU-hour.
• Standards: no international standard defining LLM inference latency or throughput was found. ISO/IEC 22989 is a terminology standard.

10. How to read evaluations and standards

Inference performance is evaluated by combining four things: (1) latency (TTFT and TPOT), (2) the throughput achievable within latency limits, (3) whether output quality is preserved, and (4) power. The most widely used public framework is MLCommons' MLPerf Inference, which defines server and offline scenarios that imitate user arrivals and sets, per LLM benchmark, TTFT and TPOT limits and an accuracy target.Sourced[Ref. 14] As of 15 October 2026 the MLCommons benchmark page shows MLPerf Inference v6.1,Sourced[Ref. 15] and the 16 September 2026 announcement reports 30 submitting organizations and new tests including an end-to-end RAG benchmark.Sourced[Ref. 15]

Category tags appear in each cell. The MLPerf latency limits are transcribed from the rules document on GitHub (master as of 15 October 2026) and change between versions. The arrangement of the table is our own.

Three points on how to read these.

• Numbers depend on settings and who measured: MLPerf has submitters measure with their own optimizations, but the rules fix the harness, accuracy targets and even whether speculative decoding is allowed. A factor measured outside such rules is a different thing.
• Benchmarks get replaced: MLPerf Inference changes its models and scenarios by version (for example, v6.1 added an end-to-end RAG test and an edge agentic test). Do not treat an old round's numbers as current.
• Standards mostly cover terminology: within this article's research no international standard setting speed or latency requirements for LLM inference was found. ISO/IEC 22989 covers concepts and terminology and does not decide pass or fail on performance. Performance claims depend on the rules and conditions of public benchmarks.

11. Glossary

12. References

1. Kwon W. et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP '23 (2023), DOI 10.1145/3600006.3613165, arXiv 2309.06180 — arxiv.org/abs/2309.06180
2. Yu G.-I. et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI '22 (2022) — usenix.org/conference/osdi22/presentation/yu
3. Leviathan Y. et al. "Fast Inference from Transformers via Speculative Decoding", ICML 2023, PMLR 202 (2023), arXiv 2211.17192 — proceedings.mlr.press/v202/leviathan23a.html
4. Chen C. et al. "Accelerating Large Language Model Decoding with Speculative Sampling", arXiv 2302.01318 (2023) — arxiv.org/abs/2302.01318
5. Dao T. et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", NeurIPS 2022, arXiv 2205.14135 — arxiv.org/abs/2205.14135
6. Ainslie J. et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023, DOI 10.18653/v1/2023.emnlp-main.298, arXiv 2305.13245 — arxiv.org/abs/2305.13245
7. Grattafiori A. et al. (Meta) "The Llama 3 Herd of Models", arXiv 2407.21783 (2024) (Table 3: the 8B model has 32 layers, model dimension 4,096, 32 heads and 8 key/value heads) — arxiv.org/abs/2407.21783
8. Patel P. et al. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting", ISCA 2024, DOI 10.1109/ISCA59077.2024.00019, arXiv 2311.18677 — arxiv.org/abs/2311.18677
9. Zhong Y. et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", OSDI '24 (2024), arXiv 2401.09670 — usenix.org/conference/osdi24/presentation/zhong-yinmin
10. Agrawal A. et al. "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve", OSDI '24 (2024), arXiv 2403.02310 — usenix.org/conference/osdi24/presentation/agrawal
11. Qin R. et al. (Moonshot AI and others) "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving", arXiv 2407.00079 (2024) (the FAST '25 version is titled "Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot") — arxiv.org/abs/2407.00079
12. Zheng L. et al. "SGLang: Efficient Execution of Structured Language Model Programs", arXiv 2312.07104 (2023) — arxiv.org/abs/2312.07104
13. NVIDIA "NVIDIA H100 Tensor Core GPU" (H100 SXM: 80 GB GPU memory, 3.35 TB/s memory bandwidth, Tensor Core figures stated with sparsity, max TDP up to 700 W; checked 15 October 2026) — nvidia.com/en-us/data-center/h100/
14. MLCommons "MLPerf Inference Rules" (inference_rules.adoc in the inference_policies repository; master as checked on 15 October 2026. Definitions of TTFT/TPOT, scenarios, accuracy targets and latency limits of the LLM benchmarks, and the speculative-decoding appendix) — github.com/mlcommons/inference_policies/blob/master/infer...
15. MLCommons "MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results" (announcement of 16 September 2026; syndicated GlobeNewswire page), and the benchmark page https://mlcommons.org/benchmarks/inference-datacenter/ (v6.1 shown; checked 15 October 2026) — financialcontent.com/article/gnwcq-2026-9-16-mlcommons-se...
16. Reddi V. J. et al. "MLPerf Inference Benchmark", ISCA 2020, DOI 10.1109/ISCA45697.2020.00045, arXiv 1911.02549 — arxiv.org/abs/1911.02549
17. Tschand A. et al. "MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI", arXiv 2410.12032 (2024) — arxiv.org/abs/2410.12032
18. ISO/IEC "ISO/IEC 22989:2022 Information technology — Artificial intelligence — Artificial intelligence concepts and terminology" (Edition 1, July 2022; JTC 1/SC 42) — iso.org/standard/74296.html
19. Li Y. et al. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", ICML 2024, arXiv 2401.15077 — arxiv.org/abs/2401.15077
20. vLLM project GitHub repository vllm-project/vllm (Apache-2.0 license; update seen on 14 October 2026) — github.com/vllm-project/vllm

13. Claim-to-source audit

Last updated: 15 October 2026. Sources are limited to primary material (peer-reviewed and conference papers, MLCommons rules and announcements, an NVIDIA datasheet, and the ISO page). Real measurements, providers' prices and any international standard defining LLM inference latency could not be confirmed and are not given. The factors are condition-specific reports by each paper's authors, and our calculations are idealized estimates. Figures 2, 5 and 7 are vector drawings; Figures 3, 4 and 6 include our calculations; Figures 1 and 8 are AI-image placeholders only.