1. What a "score" is made of (the short version)

• A score is a composite of question set, run settings, scoring, aggregation and reporting party: NIST's initial public draft AI 800-2 (January 2026) organizes automated benchmark evaluation into three stages and nine practices: define the objectives and select benchmarks, implement and run the evaluation, and analyze and report the results. It also says that if the aim is comparison, the more consistent the evaluation settings across models, the more comparable the resultsSourced[Ref. 33]. Note that this is a draft of voluntary guidance.
• Benchmarks have a life span: The latest LLMs score above 90% on MMLU, which led to the harder HLE [Ref. 15]. An estimated 6.49% of MMLU questions contain errors [Ref. 3], and in February 2026 OpenAI stopped reporting SWE-bench Verified scores, citing contamination among other reasons [Ref. 10]Sourced.
• A score without an error bar cannot be read: For a hypothetical 30-question set at 80% accuracy, item sampling alone gives a 95% interval of about ±14 pointsOur calculation (Chapter 5). Leaderboards based on human votes, such as Arena, also need to be read with their method and intervals in mind (Chapter 8).

2. Types of evaluation and the pipeline — where deviations creep in

(1) Four types of evaluation

Evaluations of language models can be organized into four types by how they are scored (our classification).

• Matching against the answer key: MMLU (multiple choice), GPQA (multiple choice), GSM8K, MATH and AIME (matching the final answer) and similar [Ref. 1, 4, 6, 7].
• Run and score: As in HumanEval, generated code is run against unit tests to decide pass or fail. SWE-bench judges whether a fix to an issue in a real repository passes the tests [Ref. 5, 8].
• Judgement by humans or LLMs: MT-Bench, Chatbot Arena (human preference votes), parts of HELM Capabilities (LLM judges) and similar [Ref. 23, 24, 29].
• Updating benchmarks: LiveCodeBench, LiveBench and others, which add new problems continuously to avoid contamination [Ref. 13, 14].

Separately, there is a type that measures safety (suppression of harmful output); MLCommons' AILuminate evaluates the resistance of a system across 12 hazard categories [Ref. 30]. This article focuses on how to read capability evaluations; agent and safety evaluation are covered in our explainers on agent evaluation and safety and on AI safety evaluation and risk management.

(2) The pipeline, and where deviations enter

The NIST AI 800-2 initial public draft divides evaluation settings into four kinds: inference settings (such as temperature or reasoning effort), scaffolding settings (such as tool availability and best-of-N), task settings and scoring settings [Ref. 33]. Borrowing that grouping, Fig. 2 puts the whole pipeline on one page.

View original SVG, values and source annotations

3. Original papers and settings of representative benchmarks

The table below gives the scale and format confirmed in the original papers and official pages of the main benchmarks named in this article. It does not give score levels. The same name can denote different versions or subsets, so always check "which version".

All Sourced (original papers, official pages and journals; checked 15 October 2026). Part of the "cautions in reading" column is our commentary.

4. Settings move scores — format, pass@k, thinking strength, serving

Even the same model gives different scores under different evaluation settings. The table below organizes the main factors confirmed in primary sources.

All Sourced (original papers and the NIST draft). The right-hand column is our commentary. NIST AI 800-2 is an initial public draft (January 2026), not a final version.

pass@k as an example: the same model can be "1%" or "75%"

For a model with about 1% per-sample success (2 of n = 200 samples pass), the HumanEval unbiased estimator gives pass@1 of 1.0%, pass@10 of 9.8% and pass@100 of 75.1%. At 5% success (c = 10), pass@1 is 5.0%, pass@10 is 40.9% and pass@100 is 99.9%Our calculation. All are computed from the same results of the same model; only k differs.

5. Estimating error — item counts and confidence intervals (our calculation)

Miller treats an evaluation as an experiment whose questions can be seen as drawn from an unseen population, and gives formulas for analyzing evaluation data, measuring the difference between two models, and planning an evaluation [Ref. 21]. NIST's draft likewise says statistics should be reported with estimated uncertainty wherever possible, and gives as examples of sources of variation the nondeterministic sampling of completions for each item and the sampling of items into the benchmark [Ref. 33]. BetterBench reports that most of the 24 benchmarks it assessed did not report statistical significance and did not allow results to be easily replicated [Ref. 22]Sourced.

Here we compute the 95% interval of accuracy for each benchmark size in the simplest case (each item right or wrong, items drawn independently). The accuracies are assumptions and not real model values.

Our calculation (standard error = √(p(1−p)/n)). Item counts are Sourced from the original papers and official pages [Ref. 4, 5, 8, 9, 1]. Accuracies are assumed. The 30 is an assumption from AIME being two exams a year (the MAA page mentions I and II) of 15 questions each; the number of items actually used must be checked for each evaluation.

When two models are compared on the same question set, an analysis that uses the item-by-item differences (a paired analysis) is effective, and NIST's draft gives a statistical test on the paired difference as the example for comparing two models' mean scores [Ref. 33]. Treating them naively as independent, at 500 items and 70% accuracy the standard error of the difference between two models is about 2.9 points and the 95% interval about ±5.7 pointsOur calculation. At 198 items it is about ±9.0 points. In other words, a difference of a few points can lie within the error, depending on the size of the question set.

Around 70% accuracy, the number of items needed to detect a difference between two independent groups, in the sense that the 95% interval around the observed difference excludes zero, is about 160 for a 10-point difference, about 650 for 5 points and about 4,000 for 2 pointsOur calculation (normal approximation, (1.96/difference)² × 2p(1−p)). This is a guide for designing your own evaluation in Chapter 9.

6. Contamination, errors and saturation — the "life" of a benchmark

(1) Three problems

Contamination is the mixing of evaluation questions or answers into training data. A survey calls it Benchmark Data Contamination and describes it as a model inadvertently incorporating evaluation-benchmark information from its training data, leading to inaccurate or unreliable performance at evaluation [Ref. 18]. Errors are mistakes in the questions or answer keys themselves, and saturation is when top models reach such high scores that differences disappear (our organization).

(2) Evidence of contamination

• GSM1k: Zhang and colleagues commissioned a new set of arithmetic problems, GSM1k, mirroring the style and difficulty of GSM8K, and evaluated leading open and closed models. Accuracy dropped by up to 8%, and several families of models showed evidence of systematic overfitting across almost all model sizes. They also report that many models, especially frontier ones, show minimal signs of overfitting, and that all models broadly generalize to novel problemsSourced[Ref. 16].
• Rephrasing: Yang and colleagues showed that removal based on n-gram overlap is insufficient, because simple variations such as paraphrasing or translation bypass it, and that if such variations are not removed even a 13B model can overfit and reach scores on par with GPT-4. They also report that 8 to 18% of HumanEval overlaps with the pre-training data of RedPajama-Data-1T and StarCoder-DataSourced[Ref. 17].

(3) Evidence of errors and saturation

• Errors: Gema and colleagues re-annotated 5,700 MMLU questions by hand and estimated that 6.49% of MMLU questions contain errors, with 57% of the Virology questions analyzed containing errors, and showed discrepancies with the originally reported performance [Ref. 3].
• Saturation: The HLE paper says its motivation was that the latest LLMs now exceed 90% on popular benchmarks such as MMLU, which limits measurement of the state of the art [Ref. 15]. OpenAI also wrote that recent frontier models score very highly on MATH and GSM8K, so these benchmarks no longer differentiate models effectively [Ref. 44].

(4) A case: SWE-bench and its successors

SWE-bench is an example where the "life" of a benchmark can be followed along one lineage.

• October 2023: SWE-bench consisted of 2,294 real GitHub issues from 12 Python repositories, and the best model at release (Claude 2) solved 1.96% of them [Ref. 8].
• 13 August 2024: OpenAI, judging that SWE-bench contained tasks that were hard or impossible to solve and systematically underestimated model capability, released SWE-bench Verified, a 500-problem subset confirmed by experts. In the annotation, 38.3% of tasks were flagged as having underspecified problem statements and 61.1% as having unit tests that could reject valid solutions, and 68.3% were excluded overall. OpenAI also said this filtering was likely overly aggressive [Ref. 9].
• September 2025: SWE-Bench Pro was released with 1,865 problems from 41 repositories, split into public, held-out and commercial sets (arXiv 2509.16941) [Ref. 11].
• 23 February 2026: OpenAI announced that it had stopped reporting SWE-bench Verified scores, recommended that other model developers do the same, and recommended reporting SWE-bench Pro results. Its reasons were that state-of-the-art progress had slowed from 74.9% to 80.9% over the previous six months; that an audit of 27.6% of the problems the models often failed to solve (138 problems) found material problems, such as test cases that reject functionally correct submissions, in at least 59.4%; and that every frontier model tested could reproduce the original fix (the gold patch) or specifics of the problem statement. It wrote that improvements on Verified increasingly reflect how much the model was exposed to the benchmark during training rather than real-world software development abilitySourced[Ref. 10].
• 8 September 2026: Other authors claimed that SWE-Bench Pro also has reward hacking through leakage of gold solutions or hidden evaluation information, and task-quality problems such as misleading problem statements and improperly scoped tests, and published a paper proposing a verified version, SWE-Bench Pro VerifiedNot yet confirmed[Ref. 12] (a pre-peer-review preprint whose content we have not verified).

(5) Countermeasures, and their limits

• Updating benchmarks: LiveCodeBench separates problems by release date and LiveBench updates its questions monthly and scores automatically against objective answers [Ref. 13, 14]. But the model's training cutoff must be matched against the problem period (commentary).
• Held-out sets: HLE keeps a private held-out set apart from the public set, to check overfitting and gaming of the public benchmark [Ref. 15].
• Canary strings: NIST's draft lists mixing unique strings into benchmark data, to check for blatant training on the test set, as an emerging practice. It also lists keeping benchmark data hidden during training, and using data created after the system was built, as ways to lower contamination risk [Ref. 33].
• Selection criteria: HELM Capabilities says it chose scenarios by considering whether they are saturated given state-of-the-art performance, how recent they are, and their quality (clarity, adoption and reproducibility) [Ref. 29].

7. Who measured it — the evaluator

Even for the same benchmark, the freedom in settings, the interests involved and the verifiability differ with who ran it. The table below organizes the evaluators confirmed in primary sources. It does not say which is "right"; it shows what to check.

All Sourced (papers and official pages; checked 15 October 2026). The right-hand column is our commentary. The latest HELM Capabilities release is the version and date shown on the page.

8. LLM-as-judge, human evaluation and leaderboards (such as LMArena): methods and limits

(1) LLM-as-judge (scoring by an LLM)

Zheng and colleagues examined using strong LLMs as judges to evaluate open-ended responses. They showed that judges have position bias (favoring an answer by where it appears), verbosity bias and self-enhancement bias (favoring their own outputs), as well as limited reasoning ability, and proposed mitigations. They then reported that strong judges such as GPT-4 reach more than 80% agreement with both controlled and crowdsourced human preferences, the same level of agreement as between humansSourced[Ref. 23].

HELM Capabilities is a real example of this method. For WildBench, several LLMs (GPT-4o, Llama 3.1 405B Instruct Turbo and Claude 3.5 Sonnet) score a response against a checklist and the scores are averaged. For Omni-MATH judging, HELM did not use the judging prompt of the original paper but switched to the LLM-judge prompt of HELM Safety, because human verification (canary results) showed that the original prompt encouraged hallucination by the judgesSourced[Ref. 29]. This is primary-source evidence that the choice of judges and the prompt can change a score (commentary).

(2) Human evaluation and leaderboards: the case of Arena

Chatbot Arena is a mechanism in which users compare the anonymous responses of two models and vote. The paper describes using pairwise comparison and crowdsourced votes, ranking with a Bradley–Terry model, collecting more than 240K votes at the time of the paper, and finding that votes agree well with expert raters [Ref. 24]. In the Bradley–Terry model, the probability that model m beats m′ is 1/(1 + e^{ξ_m′−ξ_m}), where ξ is each model's coefficient [Ref. 24]. From this formula we calculated the relation between the coefficient difference and the win probability.

Our calculation (formula: the Bradley–Terry model of the Chatbot Arena paper [Ref. 24]). We have not confirmed the scale of the score shown on the site (the coefficient itself or a converted value), so check the official explanation for how a difference in the displayed score maps to a win probability.

In other words, a rank separated by a small coefficient difference is close to a coin flip in any single comparison. Arena's official policy (updated 1 September 2026) says the evaluation and ranking pipelines are open-sourced as Arena-Rank so anyone can audit them with released data. It lists only generally available models (open weights, public APIs, public services); collects at least 1,000 votes before a rating is considered stable; treats scores of models evaluated before release as preliminary until enough fresh votes are collected after release; allows providers to test several variants before release; and says that even when sampling probabilities are uneven, the regression uses reweighting so the scores stay unbiasedSourced[Ref. 26].

(3) The dispute over fairness of leaderboards: both sides

"The Leaderboard Illusion" by Singh and colleagues claimed that undisclosed private-testing practices favor a few providers who can test several variants before release and withdraw scores if they wish, and that because they can choose the best score, Arena scores are biased by selective disclosure. It also estimated that proprietary closed models are sampled at higher rates than open-weight models, creating asymmetries in access to Arena data, and said that access to Arena data yields large gains on the Arena distribution [Ref. 25]. In a statement of 9 May 2025 (updated 13 June 2025), Arena responded that the paper contains constructive points but also incorrect claims. For example, it said that the plot behind the claim that pre-release testing could raise scores by more than 100 points was a simulation with Gaussians unrelated to Arena's policy or real performance, that the real effect is small (about +11 Elo after about 50 tests and 3,000 votes) and vanishes as fresh data accumulate, and that open models account for 40.9% in its official statistics [Ref. 27]. This article does not judge which is right. What is certain is that both sides are publicly arguing that how a leaderboard is operated (who may test which model, when and how many times) can affect what a score means, and that Arena has written down its policy of treating pre-release scores as preliminary (commentary).

9. A materials engineer's view — evaluating on your own problems

A procedure for your own evaluation (commentary)

1. Decide the purpose: write down what the results will be used for (narrowing candidates, an adoption decision, monitoring in operation) and the capability you want to measure.
2. Build the evaluation set: have the responsible person fix representative questions and answers. Give it a spread of difficulty and recheck for errors.
3. Fix the settings: decide and record the prompt, shots, temperature, thinking strength, tools and scoring method.
4. Run several times and give the error: run several times under the same settings and add an interval suited to the number of items.
5. Keep results and resources used: keep the outputs, tokens used, cost, and the model version and date.
6. Re-evaluate regularly: prepare for model updates and retirements by being able to rerun under the same settings.

10. Limits, and what could not be confirmed

(1) Current values of scores

This article does not list the scores or rankings of particular models. The latest rankings of HELM, Arena and others change after the check date. The latest release of HELM Capabilities we confirmed on its page was v1.15.0 (24 November 2025), and we did not check whether a newer one had appeared by 15 October 2026.

(2) Verification of the claims of companies and institutions

OpenAI's analysis of SWE-bench Verified (the audit and the evidence of contamination), the claims of both Arena and "The Leaderboard Illusion", and the claims of the SWE-Bench Pro Verified paper are all statements in published documents, and we have not reproduced them independently. SWE-Bench Pro Verified is a pre-peer-review preprint.

(3) Text of standards and drafts

For ISO/IEC standards we checked only the title, publication date and scope (abstract) on the official page, not the full text. For NIST AI 800-2 we retrieved the PDF of the initial public draft and confirmed its structure and main practices, but could not confirm a final version as of 15 October 2026. NIST AI 100-1 was checked in its PDF.

(4) Scale of Arena scores

We could not confirm whether the score shown on Arena's site is the Bradley–Terry coefficient itself or a converted value. The win-probability table in Chapter 8 is for differences in the coefficient.

(5) Evaluation in materials and chemistry

We could not confirm a primary source for a widely agreed evaluation standard specific to the materials and process field, or for the effectiveness of the in-house evaluation procedure proposed here. Chapter 9 is this article's commentary.

11. How to read evaluations and standards

Because this article is itself about how to read evaluations, this chapter lists, as far as we could confirm them, the standards, guidelines and frameworks that bear on the design, running and reporting of benchmarks. Few international standards directly define the evaluation of AI; most international documents are in the form of technical reports (TR) and technical specifications (TS) that organize concepts and terminology. The original papers of individual benchmarks are in Chapter 3.

Standard numbers, titles, publication dates and stages were checked on the official ISO and NIST pages (Sourced; checked 15 October 2026). The essentials of benchmarks and frameworks come from each original paper or official page. The arrangement of the table and the "cautions in reading" column are Our calculation (our own organization).

Cautions in reading

• Benchmark values change with the settings and the evaluator. The format alone can move a score by up to 76 points, and a different k can make the same model 1% or 75% (Chapter 4). Read a score as a five-part set: version, settings, scoring, item count and evaluator.
• Watch for saturation and contamination. As the examples of MMLU, GSM8K and SWE-bench Verified show, widely used question sets lose the power to separate models through errors, contamination and saturation (Chapter 6). Prefer scores that come with countermeasures such as being recent, updated or held out.
• Read standards and guidelines by whether they recommend or require, and whether they have legal force. NIST AI 800-2 is a draft of voluntary guidance, AI 100-1 is a voluntary framework, and ISO/IEC TRs and TSs are reports and specifications. HELM, MLCommons and Arena are not standards. None of them decides "which model is better".
• Distinguish declared values from measured values. A score announced by a developer is a declared value; a score you measure under your own settings is a measured value (Chapter 9).

12. Glossary

13. References (primary sources)

1. Hendrycks D. et al. "Measuring Massive Multitask Language Understanding", arXiv:2009.03300 (2020; v3 January 2021) — arxiv.org/abs/2009.03300
2. Wang Y. et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark", arXiv:2406.01574 (2024; v6 November 2024) — arxiv.org/abs/2406.01574
3. Gema A. P. et al. "Are We Done with MMLU?", arXiv:2406.04127 (2024; v3 January 2025; MMLU-Redux) — arxiv.org/abs/2406.04127
4. Rein D. et al. "GPQA: A Graduate-Level Google-Proof Q&A Benchmark", arXiv:2311.12022 (November 2023) — arxiv.org/abs/2311.12022
5. Chen M. et al. "Evaluating Large Language Models Trained on Code", arXiv:2107.03374 (July 2021; HumanEval and pass@k) — arxiv.org/abs/2107.03374
6. Cobbe K. et al. "Training Verifiers to Solve Math Word Problems", arXiv:2110.14168 (October 2021; GSM8K) — arxiv.org/abs/2110.14168
7. Hendrycks D. et al. "Measuring Mathematical Problem Solving With the MATH Dataset", arXiv:2103.03874 (March 2021) — arxiv.org/abs/2103.03874
8. Jimenez C. E. et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770 (October 2023; v3 November 2024) — arxiv.org/abs/2310.06770
9. OpenAI "Introducing SWE-bench Verified" (13 August 2024; updated 24 February 2025) — openai.com/index/introducing-swe-bench-verified/
10. OpenAI "Why we no longer evaluate SWE-bench Verified" (23 February 2026) — openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
11. Deng X. et al. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", arXiv:2509.16941 (September 2025; v2 November 2025) — arxiv.org/abs/2509.16941
12. Zheng P. et al. "SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents", arXiv:2609.08149 (8 September 2026; v3 9 October 2026; pre-peer-review preprint) — arxiv.org/abs/2609.08149
13. Jain N. et al. "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code", arXiv:2403.07974 (March 2024; v2 June 2024) — arxiv.org/abs/2403.07974
14. White C. et al. "LiveBench: A Challenging, Contamination-Limited LLM Benchmark", arXiv:2406.19314 (June 2024; v2 April 2025) — arxiv.org/abs/2406.19314
15. Phan L. et al. "A benchmark of expert-level academic questions to assess AI capabilities (Humanity's Last Exam)", Nature 649, 1139–1146 (published 28 January 2026), DOI 10.1038/s41586-025-09962-4. Preprint arXiv:2501.14249 — doi.org/10.1038/s41586-025-09962-4
16. Zhang H. et al. "A Careful Examination of Large Language Model Performance on Grade School Arithmetic", arXiv:2405.00332 (May 2024; v4 November 2024; GSM1k) — arxiv.org/abs/2405.00332
17. Yang S. et al. "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples", arXiv:2311.04850 (November 2023) — arxiv.org/abs/2311.04850
18. Xu C. et al. "Benchmark Data Contamination of Large Language Models: A Survey", arXiv:2406.04244 (June 2024) — arxiv.org/abs/2406.04244
19. Sclar M. et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting", arXiv:2310.11324 (October 2023; v2 July 2024) — arxiv.org/abs/2310.11324
20. Biderman S. et al. "Lessons from the Trenches on Reproducible Evaluation of Language Models", arXiv:2405.14782 (May 2024; v3 May 2026) — arxiv.org/abs/2405.14782
21. Miller E. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations", arXiv:2411.00640 (November 2024) — arxiv.org/abs/2411.00640
22. Reuel A. et al. "BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices", arXiv:2411.12990 (November 2024) — arxiv.org/abs/2411.12990
23. Zheng L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv:2306.05685 (June 2023; v4 December 2023) — arxiv.org/abs/2306.05685
24. Chiang W.-L. et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference", arXiv:2403.04132 (March 2024) — arxiv.org/abs/2403.04132
25. Singh S. et al. "The Leaderboard Illusion", arXiv:2504.20879 (April 2025; v2 May 2025) — arxiv.org/abs/2504.20879
26. Arena "Arena Leaderboard Policy" (official policy; published 2 March 2024, updated 1 September 2026; checked 15 October 2026) — arena.ai/blog/policy/
27. Arena "Our Response to 'The Leaderboard Illusion' Writeup" (9 May 2025, updated 13 June 2025; checked 15 October 2026) — arena.ai/blog/our-response
28. Liang P. et al. "Holistic Evaluation of Language Models", arXiv:2211.09110 (November 2022; v2 October 2023; TMLR 2023) — arxiv.org/abs/2211.09110
29. Stanford CRFM "HELM Capabilities: Evaluating LMs Capability by Capability" (blog post, 20 March 2025) — crfm.stanford.edu/2025/03/20/helm-capabilities.html
30. Ghosh S. et al. (MLCommons) "AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons", arXiv:2503.05731 (February 2025; v2 April 2025) — arxiv.org/abs/2503.05731
31. MLCommons "AILuminate" (official page; checked 15 October 2026) — mlcommons.org/ailuminate/
32. UK AI Security Institute / Meridian Labs "Inspect: An open-source framework for large language model evaluations" (official documentation; checked 15 October 2026) — inspect.aisi.org.uk/
33. NIST (CAISI) "Practices for Automated Benchmark Evaluations of Language Models", NIST AI 800-2 Initial Public Draft (January 2026), DOI 10.6028/NIST.AI.800-2.ipd — nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf
34. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1 (January 2023), DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1
35. NIST "Towards Best Practices for Automated Benchmark Evaluations" (published 30 January 2026; comments until 31 March 2026). The NIST Guidelines page (updated 30 September 2026) was also checked — www.nist.gov/news-events/news/2026/01/towards-best-practices-automated-benchmark-evaluations
36. ISO/IEC "ISO/IEC TR 42106:2026 Differentiated benchmarking of AI system quality characteristics" (July 2026; technical report) — www.iso.org/standard/42106
37. ISO/IEC "ISO/IEC TS 4213:2022 Information technology — Artificial intelligence — Assessment of machine learning classification performance" (October 2022) — www.iso.org/standard/79799.html
38. ISO/IEC "ISO/IEC TS 42119-2:2025 Artificial intelligence — Testing of AI — Part 2: Overview of testing AI systems" (November 2025) — www.iso.org/standard/84127.html
39. ISO/IEC "ISO/IEC TS 25058:2024 … SQuaRE — Guidance for quality evaluation of artificial intelligence (AI) systems" (January 2024) — www.iso.org/standard/82570.html
40. ISO/IEC "ISO/IEC 25059:2023 … SQuaRE — Quality model for AI systems" (June 2023) — www.iso.org/standard/80655.html
41. ISO/IEC "ISO/IEC 5259-2:2024 Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 2: Data quality measures" (November 2024) — www.iso.org/standard/81860.html
42. Mirza A. et al. "Are large language models superhuman chemists?", arXiv:2404.01475 (April 2024; v2 November 2024; ChemBench) — arxiv.org/abs/2404.01475
43. Mathematical Association of America (MAA) "MAA Invitational Competitions" (description of AIME; checked 15 October 2026) — maa.org/maa-invitational-competitions/
44. OpenAI "Learning to reason with LLMs" (12 September 2024; states that MATH and GSM8K no longer differentiate models) — openai.com/index/learning-to-reason-with-llms/
45. Stanford CRFM "Holistic Evaluation of Language Models (HELM)" (official site and leaderboard list; latest Capabilities release v1.15.0, 24 November 2025; checked 15 October 2026) — crfm.stanford.edu/helm/

14. Claim-to-source audit

Last updated: 15 October 2026 (check date). Sources are limited to primary materials: original papers, the Nature paper, the official pages of developers and operators, and the official NIST and ISO pages. The full text of the standards is paid and was not checked. How leaderboards and benchmarks are operated changes quickly, and we do not guarantee the state after the check date. The scale of Arena's displayed score, any newer release of HELM Capabilities, a final version of NIST AI 800-2, a widely agreed evaluation standard in materials, the effectiveness of the in-house procedure, verification of the SWE-Bench Pro Verified content, and the full text of standards could not be confirmed and are not stated. Figures in the text belong to a particular version, settings and evaluator and do not show which model is better. The error calculations rest on illustrative assumptions. Figs. 2 to 5 are vector drawings; Figs. 1 and 6 are ImageGen-generated conceptual illustrations. None of them shows a real measurement screen or product.