1. What is a training cluster? (in three points)

• The model does not fit on one machine: Narayanan et al. explain that GPU memory is limited, so large models cannot fit even on a multi-GPU server, and the number of operations required would give unrealistically long training times Sourced [Ref. 15]. So the model and the data are split across many GPUs.
• Splitting requires communication: the pieces exchange results at every step. Llama 3 405B was trained on up to 16K (16,384) H100 GPUs, with the eight GPUs of a server joined by NVLink and servers connected over a 400 Gbps network Sourced [Ref. 1].
• Keeping it running is hard: the same paper reports 466 interruptions in 54 days, 419 of them unexpected, because training proceeds synchronously and one GPU failure can require restarting the whole job Sourced [Ref. 1].

2. Parallelism: data, tensor and pipeline

Parallelism comes in kinds according to what is split. The table summarizes representative papers and published configurations.

All Sourced (the papers). The Llama 3 combination is "4D parallelism" (TP, CP, PP, DP), and the outer the dimension in the order [TP, CP, PP, DP], the slower the link it is placed on [Ref. 1].

3. Scale-up and scale-out: NVLink, InfiniBand and Ethernet

View original SVG, values and source annotations

Communication paths are described in two layers: scale-up (a small number of GPUs wired together very widely) and scale-out (many servers joined by a network). The UALink Consortium says UALink addresses efficient scale-up architectures and the UEC primarily scale-out, and that the two are complementary Sourced [Ref. 24].

All Sourced (publications by manufacturers and bodies). The list mixes finalized standards (IEEE), consortium specifications (UEC, UALink, IBTA) and vendor technology (NVLink). This article did not check product implementations or interoperability track records.

Llama 3 says it made both its model-parallelism methods and its job scheduler aware of network topology (which pods a job crosses). Knowing the tiers of the network and placing the communication-heavy parallelism on the inside is the basic design rule [Ref. 1] (this article's organization). Optical interconnects, CPO and optical transceivers are covered in our explainers on optical interconnects and SerDes, co-packaged optics, and optical transceivers.

4. Collective communication: all-reduce and estimating traffic

Training uses collective communication: all-reduce, which sums everyone's gradients and returns the result to everyone; reduce-scatter, which sums and leaves each rank with a piece; all-gather, which collects pieces onto everyone; and all-to-all, in which everyone sends a different piece to everyone. The documentation of NVIDIA's communication library NCCL lists AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, AlltoAll and others as its collective operations Sourced [Ref. 31].

Ring all-reduce cuts the data into n pieces and keeps passing them to the neighbor around a ring. Patarasuk and Yuan showed that this is bandwidth-optimal (under the cluster conditions they assume) [Ref. 6]. The NCCL-tests documentation writes the best-case time of an all-reduce of total size S, with per-rank bandwidth B, as t = (S/B) x 2(n-1)/n (common to point-to-point-based algorithms such as ring and tree). As n grows the factor only approaches 2; traffic does not grow in proportion to the number of GPUs Sourced [Ref. 7].

A traffic estimate: synchronizing gradients in Llama 3 405B

From the published configuration, we estimate a lower bound on the time to synchronize gradients Our calculation. For the 16K-GPU, DP = 128 setting of Llama 3 405B, the paper publishes 16M tokens per batch and 400 TFLOPS per GPU (41% BF16 MFU) [Ref. 1].

• Compute time per step: approximating compute as 6 x parameters x tokens, 6 x 405 x 10⁹ x 16 x 10⁶ / (16,384 x 400 x 10¹²) = about 5.93 s.
• Size of a gradient shard: the full BF16 (2-byte) gradients are 405 x 10⁹ x 2 = 810 GB. Shared across TP x PP = 8 x 16 = 128, that is about 6.3 GB per GPU.
• Best-case lower bound for all-reduce: taking each GPU's network bandwidth as 400 Gbps = 50 GB/s, (6.3 / 50) x 2 x 127 / 128 = about 0.25 s, about 4% of the step.

This is a rough lower bound that does not overlap with compute and excludes latency, FSDP weight all-gather, and gradient communication in FP32 (the paper does the reduce-scatter in FP32) [Ref. 1]. In practice the trade-off is between tricks that hide communication behind computation (such as DeepSeek-V3's DualPipe) [Ref. 2] and whatever they cannot hide.

5. Training efficiency: how to read MFU

A widely used measure of training efficiency is MFU (Model FLOPs Utilization). The PaLM paper defines MFU as the ratio of the observed throughput (tokens per second) to the theoretical maximum throughput of a system running at peak FLOPs, counting in the numerator only the operations needed for the forward and backward passes and not rematerialization Sourced [Ref. 8]. The point of the proposal was a metric independent of implementation that allows a cleaner comparison between systems.

• Published values: PaLM 540B reached an MFU of 46.2% on 6,144 TPU v4 chips (hardware FLOPs utilization, HFU, of 57.8%); GPT-3 21.3%, Gopher 32.5% and Megatron-Turing NLG 30.2% [Ref. 8]. Llama 3 405B reached 38-43% in BF16 (43% on 8K GPUs and 41% on 16K GPUs) [Ref. 1]. MegaScale trained a 175B model on 12,288 GPUs at 55.2% MFU (1.34x Megatron-LM) [Ref. 10]. A 530B run using selective activation recomputation reached 54.2% on 2,240 A100 GPUs [Ref. 9].
• Lining up MFU values does not settle superiority: chips (TPU or GPU), precision, model structure, era and counting convention (whether attention is included) all differ. The PaLM paper itself notes differences in counting, for example that the GPT-3 value is derived from non-attention model TFLOP/s and that the MT-NLG value is given both with and without attention [Ref. 8].
• Higher MFU is better, but it never reaches 100%: communication, a slow rank, memory reads and writes, checkpoints and failure recovery all show up as losses against the peak denominator. The "effective training time" in the next chapter is another loss.
• Overlapping communication with computation: the DeepSeek-V3 report says DualPipe hides most communication behind computation and that, as long as the computation-to-communication ratio stays constant, fine-grained experts can be placed across nodes with near-zero all-to-all overhead even as the model scales [Ref. 2] (the authors' claim).

6. Failures and fault tolerance

The Llama 3 paper says that because training is synchronous, one GPU failure can lead to a restart of the whole job. For a 54-day record, it reports 466 interruptions (47 planned, such as maintenance and configuration updates, and 419 unexpected), says about 78% of the unexpected ones were attributed to confirmed or suspected hardware issues (GPU and host-component failures, silent data corruption, unplanned maintenance and the like), and says significant manual intervention was needed only three times Sourced [Ref. 1]. Even so, including automated maintenance such as firmware and Linux kernel upgrades, effective training time exceeded 90% [Ref. 1].

The "record" column is Sourced (the Llama 3 paper). In the right-hand column, the items marked Our calculation and the interpretation are this article's.

A rule of thumb for the checkpoint interval

The more often checkpoints (saves of intermediate progress) are taken, the less work is lost, but the more time goes on saving. In the first-order approximation of Young and Daly, with save time d and mean time between failures M, the optimal interval is tau ~ sqrt(2dM) [Refs. 11, 12]. Putting M = 3.1 h (11,130 s) and an assumed save time d = 60 s gives tau ~ 1,155 s (about 19 min), and the fraction of wasted time is roughly d/tau + tau/(2M), about 10% Our calculation. If d could be cut to 10 s, tau ~ 472 s (about 8 min) and the fraction about 4%. This illustrates that the faster the save, the less is wasted even when failures are frequent; d is an assumption and the formula a first-order approximation.

Problems specific to scale: power swings and temperature

• Power swings: the Llama 3 paper says that when tens of thousands of GPUs raise or lower power together (all waiting for a checkpoint or a collective to finish, or the job starting or stopping), data-center power can swing instantly by tens of megawatts, stretching the limits of the power grid [Ref. 1]. NVIDIA also says power smoothing in its Rubin rack cut 50 ms peak power by about 20% (the company's own figure) [Ref. 5]. Power and cooling are covered in our explainers on power and cooling for AI data centers and on data centers and energy storage.
• Effect of temperature: in training Llama 3 405B there was a diurnal throughput variation of 1-2% depending on time of day, which the paper attributes to higher mid-day temperatures affecting GPU dynamic voltage and frequency scaling [Ref. 1].
• The analysis by Kokolis et al.: analyzing two large Meta clusters (11 months, 4 million jobs, over 150 million A100 GPU hours), they report that large jobs are the most vulnerable to failures but small jobs are the majority and should be included in optimization objectives, and that a failure model can project mean time to failure by scale [Ref. 28].

7. A materials engineer's view: the failure breakdown, temperature and cabling

8. Limits, and what could not be confirmed

(1) Scope of what is published

Most figures are from one organization's one cluster over one period (such as Llama 3's 54 days). The configurations and failure rates of other organizations' latest large training runs could not be confirmed from published material.

(2) Direction and definition of bandwidth

For NVLink and similar figures, some sources do not state whether a value is one-way or a bidirectional total, so they cannot be compared directly with a network's 400 Gbps (a one-way line rate). The "18x" in Fig. 3 is approximate.

(3) Status of specifications

UEC 1.1 (12 October 2026) was published three days before the confirmation date, and its implementation track record was not checked. The status of IEEE P802.3dj could not be confirmed. The texts of the specifications were not read.

(4) Causes of failures

The Llama 3 table is a classification and does not show a causal link to materials or temperature. A part of the table's percentages also has an inconsistency within the paper (30.1% for faulty GPUs against 35.3% from 148 / 419).

(5) Premises of the calculations

The traffic and checkpoint calculations rest on idealizations and assumptions (d = 60 s and so on) and are not measurements. MFU settings differ, so values cannot be ranked.

9. How to read evaluations and standards

The performance of a training cluster is evaluated along five lines: (1) training speed (time to reach a target accuracy on a common task), (2) efficiency (such as MFU), (3) communication performance and conformance to standards, (4) reliability (number of interruptions, effective training time) and (5) power and cooling. Line 1 is MLCommons' MLPerf; lines 2 and 4 rest mainly on definitions and reports in papers; line 3 involves standards and specifications from IEEE, IBTA, UEC and UALink. As of the confirmation date (15 October 2026) we could not confirm a finalized international standard (ISO/IEC, IEEE) that directly targets MFU (2) or reliability (4).

The arrangement of the table is Our calculation (editing by this article). The standards' texts were not checked, so no requirements are described.

Notes on reading

• Benchmark values change with settings and evaluator. MLPerf differs in system scale, configuration and version, and MFU differs in chip, precision and counting. This article transcribes values from papers and announcements and ranks nothing.
• Saturation and contamination: MLPerf updates its tasks to match the workloads of the day (this round added MoE-model tasks) [Ref. 30]. Results on older tasks may not represent the latest training workloads.
• A "standard", a "consortium specification" and a "report" are different things. IEEE Std 802.3df-2024 is a standard, UEC, UALink and IBTA are consortium specifications, and MFU and interruption counts are reports in papers. None is itself law.
• Declared versus measured. NVLink's 1.8 TB/s or the network's 400 Gbps is a line-rate ceiling. The communication speed actually obtained in training falls with latency, congestion and software.

10. Glossary

11. References (primary sources)

1. Grattafiori A. et al. (Meta) "The Llama 3 Herd of Models", arXiv:2407.21783 — arxiv.org/abs/2407.21783
2. DeepSeek-AI "DeepSeek-V3 Technical Report", arXiv:2412.19437 — arxiv.org/abs/2412.19437
3. NVIDIA "NVIDIA HGX Platform" (specifications of HGX Rubin NVL8, B300 and B200) — www.nvidia.com/en-us/data-center/hgx/
4. NVIDIA "GB200 NVL72" — www.nvidia.com/en-us/data-center/gb200-nvl72/
5. NVIDIA "Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI", NVIDIA Technical Blog (21 July 2026) — developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/
6. Patarasuk P. and Yuan X. "Bandwidth optimal all-reduce algorithms for clusters of workstations", J. Parallel Distrib. Comput. 69, 117-124 (2009), DOI 10.1016/j.jpdc.2008.09.002 — doi.org/10.1016/j.jpdc.2008.09.002
7. NVIDIA "Performance reported by NCCL tests" (nccl-tests, GitHub) — github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md
8. Chowdhery A. et al. (Google) "PaLM: Scaling Language Modeling with Pathways", arXiv:2204.02311 — arxiv.org/abs/2204.02311
9. Korthikanti V. et al. "Reducing Activation Recomputation in Large Transformer Models", arXiv:2205.05198 — arxiv.org/abs/2205.05198
10. Jiang Z. et al. "MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs", arXiv:2402.15627 — arxiv.org/abs/2402.15627
11. Young J. W. "A first order approximation to the optimum checkpoint interval", Commun. ACM 17(9), 530-531 (1974), DOI 10.1145/361147.361115 — doi.org/10.1145/361147.361115
12. Daly J. T. "A higher order estimate of the optimum checkpoint interval for restart dumps", Future Gener. Comput. Syst. 22, 303-312 (2006), DOI 10.1016/j.future.2004.11.016 — doi.org/10.1016/j.future.2004.11.016
13. NVIDIA "H100 GPU" — www.nvidia.com/en-us/data-center/h100/
14. Shoeybi M. et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", arXiv:1909.08053 — arxiv.org/abs/1909.08053
15. Narayanan D. et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM", arXiv:2104.04473 — arxiv.org/abs/2104.04473
16. Huang Y. et al. "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism", arXiv:1811.06965 — arxiv.org/abs/1811.06965
17. Rajbhandari S. et al. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models", arXiv:1910.02054 — arxiv.org/abs/1910.02054
18. Zhao Y. et al. "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel", arXiv:2304.11277 — arxiv.org/abs/2304.11277
19. Li S. et al. "PyTorch Distributed: Experiences on Accelerating Data Parallel Training", arXiv:2006.15704 — arxiv.org/abs/2006.15704
20. Zhao C. et al. "Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures", arXiv:2505.09343 — arxiv.org/abs/2505.09343
21. Ultra Ethernet Consortium "Specification History" — ultraethernet.org/specification-history/
22. UALink Consortium "Specifications" — ualinkconsortium.org/specification/
23. UALink Consortium "UALink Consortium Releases the Ultra Accelerator Link 200G 1.0 Specification" (news release, 8 April 2025) — ualinkconsortium.org/wp-content/uploads/2025/04/UALink-1.0-Specification-PR_FINAL.pdf
24. UALink Consortium "FAQ" — ualinkconsortium.org/faq/
25. InfiniBand Trade Association "InfiniBand Trade Association" (InfiniBand and RoCE Specifications) — www.infinibandta.org/
26. IEEE 802.3 "IEEE P802.3df 400 Gb/s and 800 Gb/s Ethernet Task Force" (IEEE Std 802.3df-2024) — www.ieee802.org/3/df/index.html
27. IEEE 802.3 "IEEE P802.3dj 200 Gb/s, 400 Gb/s, 800 Gb/s, and 1.6 Tb/s Ethernet Task Force" — www.ieee802.org/3/dj/index.html
28. Kokolis A. et al. (Meta) "Revisiting Reliability in Large-Scale Machine Learning Research Clusters", arXiv:2410.21680 — arxiv.org/abs/2410.21680
29. Gangidi A. et al. (Meta) "RDMA over Ethernet for Distributed Training at Meta Scale", Proc. ACM SIGCOMM 2024, 57-70, DOI 10.1145/3651890.3672233 — doi.org/10.1145/3651890.3672233
30. MLCommons "MLCommons Releases MLPerf Training v6.0 Results" (16 June 2026) — mlcommons.org/2026/06/mlperf-training-v6-0-results/
31. NVIDIA "Collective Operations", NCCL 2.32.3 documentation — docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html
32. Liu H. et al. "Ring Attention with Blockwise Transformers for Near-Infinite Context", arXiv:2310.01889 — arxiv.org/abs/2310.01889

12. Claim-to-source audit

Last updated: 15 October 2026 / Sources are limited to primary material (papers on Llama 3, PaLM, MegaScale and DeepSeek-V3, official NVIDIA, UEC, UALink, IBTA and IEEE pages, and MLCommons announcements). The texts of the standards were not checked. This field moves fast, and versions and statuses of standards and specifications are those of the confirmation date (15 October 2026). Failure rates of the latest large training runs, comparisons with aligned bandwidth definitions, causality between failures and materials, and the texts of standards could not be confirmed and are therefore not stated. Figs. 2 to 6 are vector drawings and Figs. 1 and 7 are AI-generated images; none of them depicts a real facility, product or measurement result.