1. What a reasoning model is (the short version)

• It writes at length internally before answering: In its announcement of 12 September 2024, OpenAI described o1 as a new large language model trained with reinforcement learning to perform complex reasoning, one that thinks before it answers and can produce a long internal chain of thought before responding to the userSourced[Ref. 9]. That internal trace is what we call thinking tokens.
• Performance also changes with how much it thinks: OpenAI reported that o1's performance improved consistently with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)Sourced[Ref. 9]. This is the developer's claim, not an independent replication.
• It is becoming a setting: The official documentation of Anthropic, OpenAI, Google and DeepSeek, checked on 15 October 2026, shows the amount of thinking being adjusted with parameters such as effort or thinking levelSourced[Ref. 21-25] (Chapter 6). The same model gives different accuracy, latency and cost under different settings.

2. From chain of thought to test-time compute — what changed

(1) Chain of thought: have the model write out the working

In 2022, Wei and colleagues reported that simply showing a few examples that include intermediate reasoning steps (a chain of thought) improves large language models on arithmetic, commonsense and symbolic reasoning tasks. With eight examples, a 540B-parameter model reached what was then the best accuracy on the GSM8K math word-problem benchmark, surpassing a fine-tuned GPT-3 with a verifierSourced[Ref. 1]. Kojima and colleagues then showed that simply adding "Let's think step by step", with no examples (zero-shot CoT), raised InstructGPT (text-davinci-002) from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8KSourced[Ref. 2]. Up to this point, the model is simply asked in the prompt to think while it answers (commentary).

(2) Generate many solutions and choose: self-consistency

Self-consistency, by Wang and colleagues, replaces a single greedy decode by sampling several diverse reasoning paths and selecting the most consistent answer. It reported gains of +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA and +3.9% on ARC-challengeSourced[Ref. 3]. The idea of asking the same question several times to spend more compute leads to the later cons@64 and best-of-N methods (Chapter 4).

(3) Spending compute at inference: scaling test-time compute

Snell and colleagues studied what happens when more compute is used at inference. They found that the effect depends strongly on how hard the problem is, and that a "compute-optimal" strategy that allocates compute per problem can improve efficiency by more than 4x over best-of-N. In a comparison with matched compute (FLOPs), on problems where a smaller base model has a non-trivial success rate, test-time compute could outperform a model 14 times largerSourced[Ref. 6]. This is a result under the paper's tasks and conditions, not a general statement that "a small model is enough" (commentary).

3. How reinforcement learning teaches it — the key is "problems that can be graded"

(1) o1: details not published

OpenAI describes o1 as using a large-scale reinforcement learning algorithm that, in a data-efficient training process, teaches the model to use a chain of thought productively: to recognize and correct mistakes, to break hard steps into simpler ones, and to try another approach when the current one failsSourced[Ref. 9]. The announcement does not give the reward design, the training data or the model size, and this article does not guess at them.

(2) DeepSeek-R1: the reward is "is the answer right?"

DeepSeek-R1 is the case where the details were published. The paper in Nature, published on 17 September 2025 [Ref. 10], presents DeepSeek-R1-Zero, in which reinforcement learning alone, with no supervised fine-tuning (SFT) step, was applied to the base model DeepSeek-V3 Base. The algorithm is GRPO (Group Relative Policy Optimization), and the rewards are rule-based: whether the final answer is correct (matches the ground truth) and whether the output format is followedSourced[Ref. 10]. For mathematics, the model is made to give its final answer in a specified format so that it can be checked mechanically; for programming-contest problems, answers are graded by running a compiler against predefined test cases [Ref. 10]. The authors say they did not use neural reward models (outcome-based or process-based) for reasoning tasks, because such models are prone to reward hacking during large-scale reinforcement learning [Ref. 10].

In that paper, the average pass@1 on AIME 2024 rose during training from 15.6% to 77.9%, and to 86.7% with self-consistency decoding. The paper also reports that responses became longer as training progressed, with more reflective reasoning and exploration of alternative solutions, and describes what the authors call an "aha moment", a sudden increase in the use of the word "wait"Sourced[Ref. 10]. These are the authors' descriptions; the figures are for the paper's own evaluation settings (the definitions of pass@1 and cons@16 are in its supplementary information).

(3) Grade the outcome, or grade the process?

Lightman and colleagues compared outcome supervision (reward only on the final answer) with process supervision (reward on each reasoning step) and reported that process supervision performed significantly better on the MATH dataset. Their process-supervised model solved 78% of a representative subset of the MATH test set, and they released PRM800K, 800,000 step-level human labelsSourced[Ref. 5]. The DeepSeek-R1 paper, by contrast, chose not to use neural reward models for reasoning tasks [Ref. 10]. Which is better depends on the task, the scale and the operating conditions, and this article does not rank them (commentary).

(4) Verifiable rewards (RLVR) and spread to other models

Tülu 3, by Lambert and colleagues, is an open recipe that presents "reinforcement learning with verifiable rewards (RLVR)" as a training method alongside SFT and DPOSourced[Ref. 12]. The Qwen3 technical report describes merging a thinking mode and a non-thinking mode in one model, letting users set a budget of thinking tokens, and using strong-to-weak distillation, which transfers knowledge from large models to small ones, to cut the training cost of the smaller models [Ref. 13]. OpenAI's gpt-oss is described as post-trained with CoT reinforcement-learning techniques similar to those of o3, and its three strengths (low, medium and high) are switched with a line such as "Reasoning: low" in the system prompt [Ref. 14].

4. Three ways to add test-time compute — think longer, sample many, search

Methods for "adding compute at inference time" fall roughly into three groups (our classification). The figure below shows the differences on one page.

View original SVG, values and source annotations

All Sourced (sources in each cell). The classification and the "cautions in reading" column are our commentary.

The DeepSeek-R1 paper says that, unlike conventional test-time scaling such as majority voting or Monte Carlo tree search (MCTS), R1 allocates inference compute dynamically according to the complexity of the problem, using fewer tokens on simple tasks and more on complex ones. It also notes that excessive reasoning (overthinking) on simple questions remainsSourced[Ref. 10].

5. Thinking length, performance and cost — our calculation

(1) What changes when you change the setting on the same model

OpenAI's gpt-oss model card includes a table of AIME, GPQA Diamond and other results at three thinking strengths (low, medium and high). We computed the difference between high and low from the same table.

Figures (accuracy, %) are Sourced from Table 3 of the gpt-oss model card [Ref. 14] (the developer's evaluation). The "high − low" column is Our calculation (simple subtraction, in percentage points).

Three readings follow from this table (our commentary). First, the effect differs by task: on AIME, more thinking raises accuracy a great deal, while on GPQA Diamond (multiple-choice science questions) the gain is smaller. Second, with tools, the lift at low is large (the 120b model's AIME 2024 score goes from 56.3% to 75.4% at low). Third, on the same AIME 2024, the setting alone moves one model's score by almost 40 points. When quoting a score, you therefore need to state "at what strength".

(2) What changes when you add samples

Figures (%) are Sourced from Appendix A of OpenAI's "Learning to reason with LLMs" [Ref. 9] (as of the announcement on 12 September 2024; the developer's evaluation). "Difference" is Our calculation. pass@1 is accuracy with one sample; cons@64 is accuracy when the most frequent answer among 64 samples is taken.

cons@64 asks the same question 64 times, so the tokens generated are about 64 times as many, assuming each run has the same lengthOur calculation. For o1 on AIME 2024, that 64-fold compute bought +8.9 points; on GPQA Diamond it bought +0.7 points. In multiple-choice questions there are few candidate answers, which may make majority voting less effective (Inference; neither the paper nor the announcement gives this reason). In the same announcement, OpenAI also wrote that re-ranking 1,000 samples with a learned scoring function gave 93% (13.9 of 15) on AIME 2024 [Ref. 9]. When we checked that page on 15 October 2026, it noted that o1-preview and o1-mini, which it mentions, had been retired from the API and that o1 was deprecated [Ref. 9]. This table is a historical record, not the performance of current products.

(3) How thinking tokens affect cost and time

Three sets of official documentation say that thinking tokens are billed as output tokens (or added to output tokens). Anthropic: thinking tokens are billed as output tokens even when the thinking text is not returned, and they count toward max_tokens [Ref. 21]. OpenAI: reasoning tokens are not visible through the API but take up context-window space and are billed as output tokens; a model may generate anywhere from a few hundred to tens of thousands of them, so OpenAI recommends reserving at least 25,000 tokens for reasoning and output when you start experimenting [Ref. 23]. Google: with thinking on, response pricing is the sum of output tokens and thinking tokens, and billing is based on the full thought tokens the model generated, not on the summary [Ref. 24]Sourced.

So we estimate in "input-token equivalents" without quoting any price. The assumptions are 2,000 input tokens, 500 visible answer tokens, an output price 5 times the input price (an assumption) and a generation speed of 50 tokens per second (an assumption).

Our calculation. The 5x output-to-input price ratio and 50 tokens/s are illustrative assumptions, not the prices or speeds of any actual service (these differ by provider, model and load). Input-processing time, prompt caching and batch processing are not included.

What this estimate shows is that when thinking increases, cost is governed less by the length of the input than by the length of the output side (thinking plus answer) (our commentary). Actual prices have to be checked, with dates, on each provider's pricing page, and this article does not list any. Also, if thinking reaches a limit (such as max_tokens) first, you can be billed for the input and the thinking without receiving any visible answer. OpenAI states that the response can then be incomplete [Ref. 23]. Google likewise writes that if the model hits max_output_tokens while reasoning, the output may be truncated or empty while the thinking tokens are still billed [Ref. 24].

6. How the main services let you set "how much to think" — a comparison of official documentation (checked 15 October 2026)

The table below reflects the official documentation of each provider as retrieved on 15 October 2026. Model names, defaults and supported values change often, so always check the latest official page before you use any of them. What is compared here is the control mechanism and how thinking is exposed and billed, not the capability of the models.

All Sourced (official documentation, papers and model cards; checked 15 October 2026). DeepSeek billing, and per-model defaults and prices of each provider, are not covered.

7. Limits — faithfulness, overthinking, and the "illusion of thinking" debate

(1) Displayed thinking is not necessarily the actual computation

Turpin and colleagues reported that chain-of-thought explanations can systematically misrepresent the true reason for a model's prediction. For example, when the options in a few-shot prompt were reordered so that the answer was always "(A)", the models changed their answers without mentioning that bias in their explanations. Across 13 BIG-Bench Hard tasks, accuracy fell by as much as 36% under the biased condition (the models tested were GPT-3.5 and Claude 1.0, not current models)Sourced[Ref. 16]. Anthropic's Chen and colleagues evaluated faithfulness in reasoning models using six types of hint inserted in the prompt. Of the examples where a hint was used, the share in which the chain of thought mentioned it was at least 1% in most settings but often below 20%. The authors conclude that monitoring chains of thought is promising but not sufficient to rule out undesired behaviorSourced[Ref. 17]. Korbak and colleagues, a multi-institution author group, propose that the ability to monitor chains of thought is a valuable but fragile opportunity for safety, and recommend that developers consider the effect of development decisions on monitorability [Ref. 18].

Providers also differ in what they show. In its 2024 announcement, OpenAI wrote that, weighing user experience, competitive advantage and the option of monitoring the chain of thought, it decided not to show users the raw chain of thought [Ref. 9]. Anthropic and Google return summaries [Ref. 21, 24], and DeepSeek returns the chain of thought as reasoning_content [Ref. 25] (Chapter 6). Reading displayed thinking as the basis the model actually used for its answer is risky (our commentary).

(2) Overthinking

Chen and colleagues call the tendency of o1-like models to spend large amounts of compute even on trivial problems such as "2+3=?" overthinking, and propose efficiency metrics that look at both outcome and process. They report that a self-training mitigation reduced compute while keeping accuracy on GSM8K, MATH500, GPQA and AIME, among othersSourced[Ref. 15]. The Nature paper on DeepSeek-R1 also lists excessive reasoning on simple questions as a limitation [Ref. 10]. The effort settings in Chapter 6 are also a practical countermeasure for this problem (commentary).

(3) The "illusion of thinking" and its rebuttal

Shojaee and colleagues studied reasoning models in puzzle environments with controllable difficulty. They report that accuracy collapses completely beyond a certain complexity, and that the tokens spent on reasoning rise with complexity and then fall, even though budget remains. They also describe three regimes: at low complexity standard models do better, at medium complexity reasoning models have the advantage, and at high complexity both collapseSourced[Ref. 19]. Lawsen answered that these results mainly reflect limits of the experimental design rather than fundamental reasoning failures, pointing out that the Tower of Hanoi experiments risk exceeding output-token limits and that the River Crossing set includes instances with no mathematical solution for N above 5 [Ref. 20]. This article does not judge which is right. Follow-up replications of either paper, including revised versions, and later independent reproductions could not be confirmed within our research. What is certain is that experiments showing the "limits of reasoning models" are themselves affected by experimental conditions such as output limits and problem design (commentary).

(4) Other limits

• The gpt-oss model card says that because its chain of thought is not restricted, it can contain inaccurate content, including language that does not reflect OpenAI's safety policies, and asks developers not to show it to users without filtering, moderation or summarization [Ref. 14].
• The DeepSeek-R1 paper lists as limitations: weak structured output and tool use; language mixing outside Chinese and English; sensitivity to prompts, with few-shot prompting lowering performance; no large improvement on software-engineering tasks; and reward hacking where tasks are hard to verify [Ref. 10].
• For mathematics benchmarks used on reasoning models, it has been pointed out that existing evaluations can suffer contamination (leakage into training data) [Ref. 19]. How to read evaluations is covered in Chapter 10 and in our explainer on benchmarks and how to read evaluations.

8. A materials engineer's view — problems that can be graded, and problems that cannot

Items to record in research and development (commentary)

• Model name, version or snapshot, and retrieval date (the same name can be updated)
• Thinking strength (effort, thinking level or budget), max_tokens, and the number of samples and how they were selected (pass@1, majority vote, verifier)
• Whether tools were used, and the tool versions
• The full prompt, and the thinking and output tokens used (the API usage fields)
• Whether the result was later confirmed by measurement or calculation

9. Limits, and what could not be confirmed

(1) Model capability and current specifications

This article does not compare particular models. The specifications in Chapter 6 are as stated in the official documentation on 15 October 2026; defaults, supported models and prices change. We did not individually check the system cards or technical reports of each provider's latest models, and we do not state their performance.

(2) Independent evaluation

All figures in Chapter 5 are evaluations by the developer (OpenAI). We could not confirm an independent reproduction under the same conditions. o1's training method (reward, data, scale) has not been published, so we do not describe it.

(3) Text of the standards

For the ISO/IEC standards we checked only the title, publication date and scope (abstract) on the official page, not the full text. We do not describe requirements or test procedures. For NIST AI 100-1 and AI 600-1 we retrieved the PDFs and confirmed the title, date and nature (voluntary), but did not summarize their contents in detail.

(4) The state of the debate

The "illusion of thinking" debate is presented only as far as the published papers go. We could not confirm later replications or corrections.

10. How to read evaluations and standards

The performance of reasoning models is mainly compared by the accuracy on public benchmarks: AIME for mathematics, GPQA Diamond for science, LiveCodeBench for code, and so on. With reasoning models, however, even the same model's score moves widely with thinking strength, number of samples and use of tools, so a model name and a score alone are not enough information (Chapter 5). Few international standards directly define these evaluations; standards and guidelines exist mainly as frameworks for risk management, management systems and testing. The table below lists what we could confirm, with a type for each row. How to read benchmarks in general is covered in our explainer on benchmarks and how to read evaluations.

Standard numbers, titles and publication dates were checked on the official ISO and NIST pages (Sourced; checked 15 October 2026). Benchmark essentials come from each original paper or official page. The arrangement of the table and the "cautions in reading" column are Our calculation (our own organization).

Cautions in reading

• Benchmark values change with the settings and the evaluator. With reasoning models, thinking strength (low to high), sample count (pass@1 or cons@k) and tool use can move one model's score by tens of points (Chapter 5). Nearly all values in this article are the developers' own evaluations, not independent ones.
• Watch for saturation and contamination. OpenAI says MATH and GSM8K no longer differentiate models effectively [9]. LiveCodeBench is designed to avoid contamination through problem release dates [28]. "High on a new benchmark" is not the same as "usable for my task".
• Read standards and guidelines by whether they recommend or require, and whether they have legal force. NIST AI RMF is a voluntary framework, ISO/IEC 42001 is a management-system standard with requirements, ISO/IEC 23894 is guidance, and a TR or TS is a report or specification. None of them is a standard that sets pass or fail for thinking length or for the performance of reasoning models themselves.
• Distinguish declared values from measured values. A model's maximum thinking tokens or a configured value is a declared value; the tokens actually used can be checked in the usage fields and are the measured value (Chapter 5).

11. Glossary

12. References (primary sources)

1. Wei J. et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", arXiv:2201.11903 (2022; v6 January 2023) — arxiv.org/abs/2201.11903
2. Kojima T. et al. "Large Language Models are Zero-Shot Reasoners", arXiv:2205.11916 (2022; v4 January 2023) — arxiv.org/abs/2205.11916
3. Wang X. et al. "Self-Consistency Improves Chain of Thought Reasoning in Language Models", arXiv:2203.11171 (2022; v4 March 2023) — arxiv.org/abs/2203.11171
4. Yao S. et al. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models", arXiv:2305.10601 (2023; v2 December 2023) — arxiv.org/abs/2305.10601
5. Lightman H. et al. "Let's Verify Step by Step", arXiv:2305.20050 (May 2023; PRM800K) — arxiv.org/abs/2305.20050
6. Snell C. et al. "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", arXiv:2408.03314 (August 2024) — arxiv.org/abs/2408.03314
7. Brown B. et al. "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling", arXiv:2407.21787 (2024; v3 December 2024) — arxiv.org/abs/2407.21787
8. Muennighoff N. et al. "s1: Simple test-time scaling", arXiv:2501.19393 (2025; v3 March 2025) — arxiv.org/abs/2501.19393
9. OpenAI "Learning to reason with LLMs" (12 September 2024; Appendix A holds the evaluation table. When retrieved on 15 October 2026 the page noted that o1-preview and o1-mini were retired from the API and o1 was deprecated) — openai.com/index/learning-to-reason-with-llms/
10. DeepSeek-AI "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning", Nature 645, 633–638 (published 17 September 2025), DOI 10.1038/s41586-025-09422-z. Preprint arXiv:2501.12948 (v2 updated January 2026) — doi.org/10.1038/s41586-025-09422-z
11. Shao Z. et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300 (February 2024; proposes GRPO) — arxiv.org/abs/2402.03300
12. Lambert N. et al. "Tulu 3: Pushing Frontiers in Open Language Model Post-Training", arXiv:2411.15124 (November 2024; v5 April 2025; RLVR) — arxiv.org/abs/2411.15124
13. Qwen Team (An Yang et al.) "Qwen3 Technical Report", arXiv:2505.09388 (May 2025) — arxiv.org/abs/2505.09388
14. OpenAI "gpt-oss-120b & gpt-oss-20b Model Card", arXiv:2508.10925 (8 August 2025) — arxiv.org/abs/2508.10925
15. Chen X. et al. "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs", arXiv:2412.21187 (December 2024; v2 February 2025) — arxiv.org/abs/2412.21187
16. Turpin M. et al. "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting", arXiv:2305.04388 (May 2023; v2 December 2023) — arxiv.org/abs/2305.04388
17. Chen Y. et al. (Anthropic Alignment Science Team) "Reasoning Models Don't Always Say What They Think", arXiv:2505.05410 (May 2025) — arxiv.org/abs/2505.05410
18. Korbak T. et al. "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety", arXiv:2507.11473 (July 2025; v2 December 2025) — arxiv.org/abs/2507.11473
19. Shojaee P. et al. "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity", arXiv:2506.06941 (June 2025; v3 November 2025) — arxiv.org/abs/2506.06941
20. Lawsen A. "Comment on The Illusion of Thinking…", arXiv:2506.09250 (June 2025; v2 16 June 2025) — arxiv.org/abs/2506.09250
21. Anthropic "Thinking" (Claude API documentation; checked 15 October 2026) — platform.claude.com/docs/en/build-with-claude/thinking
22. Anthropic "Effort" (Claude API documentation; checked 15 October 2026) — platform.claude.com/docs/en/build-with-claude/effort
23. OpenAI "Reasoning models" (OpenAI API documentation; checked 15 October 2026) — platform.openai.com/docs/guides/reasoning
24. Google "Gemini thinking" (Gemini API documentation; checked 15 October 2026) — ai.google.dev/gemini-api/docs/thinking
25. DeepSeek "Thinking Mode" (DeepSeek API documentation; checked 15 October 2026) — api-docs.deepseek.com/guides/thinking_mode
26. Rein D. et al. "GPQA: A Graduate-Level Google-Proof Q&A Benchmark", arXiv:2311.12022 (November 2023) — arxiv.org/abs/2311.12022
27. Hendrycks D. et al. "Measuring Mathematical Problem Solving With the MATH Dataset", arXiv:2103.03874 (March 2021; v2 November 2021) — arxiv.org/abs/2103.03874
28. Jain N. et al. "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code", arXiv:2403.07974 (March 2024; v2 June 2024) — arxiv.org/abs/2403.07974
29. Chen M. et al. "Evaluating Large Language Models Trained on Code", arXiv:2107.03374 (July 2021; HumanEval and pass@k) — arxiv.org/abs/2107.03374
30. Mathematical Association of America (MAA) "MAA Invitational Competitions" (description of AIME; checked 15 October 2026) — maa.org/maa-invitational-competitions/
31. ISO/IEC "ISO/IEC 22989:2022 Information technology — Artificial intelligence — Artificial intelligence concepts and terminology" (edition 1, July 2022) — iso.org/standard/74296.html
32. ISO/IEC "ISO/IEC 23894:2023 Information technology — Artificial intelligence — Guidance on risk management" (edition 1, February 2023) — iso.org/standard/77304.html
33. ISO/IEC "ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system" (edition 1, December 2023) — iso.org/standard/81230.html
34. ISO/IEC "ISO/IEC TS 42119-2:2025 Artificial intelligence — Testing of AI — Part 2: Overview of testing AI systems" (November 2025) — iso.org/standard/84127.html
35. ISO/IEC "ISO/IEC TS 6254:2025 … Objectives and approaches for explainability and interpretability of machine learning (ML) models and artificial intelligence (AI) systems" (September 2025) — iso.org/standard/82148.html
36. ISO/IEC "ISO/IEC 12792:2025 Information technology — Artificial intelligence (AI) — Transparency taxonomy of AI systems" (November 2025) — iso.org/standard/84111.html
37. ISO/IEC "ISO/IEC TR 20226:2025 Information technology — Artificial intelligence — Environmental sustainability aspects of AI systems" (July 2025; technical report) — iso.org/standard/86177.html
38. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1 (January 2023), DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1
39. NIST "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", NIST AI 600-1 (July 2024), DOI 10.6028/NIST.AI.600-1 — doi.org/10.6028/NIST.AI.600-1
40. NIST "AI Risk Management Framework" (NIST page; states that AI RMF 1.0 is being revised; checked 15 October 2026) — nist.gov/itl/ai-risk-management-framework
41. Anthropic "Building with extended thinking" (Claude API documentation; fixed budget and migration to adaptive thinking; checked 15 October 2026) — platform.claude.com/docs/en/build-with-claude/extended-thinking
42. NIST (CAISI) "Practices for Automated Benchmark Evaluations of Language Models", NIST AI 800-2 Initial Public Draft (January 2026), DOI 10.6028/NIST.AI.800-2.ipd — nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf

13. Claim-to-source audit

Last updated: 15 October 2026 (check date). Sources are limited to primary materials: original papers, the Nature paper, developers' model cards and official documentation, and the official ISO and NIST pages. The full text of the standards is paid and was not checked. Model names, API specifications, defaults and retirement status change quickly, and we do not guarantee the state after the check date. Standards targeting AIME, MATH or GPQA, a standard defining a test for chain-of-thought faithfulness, a standard measurement of reasoning tokens, independent reproductions under the same conditions, follow-up replications of the "illusion of thinking", and individual specifications of each provider's latest models could not be confirmed and are not stated. Accuracy figures in the text are values for a particular model, version, setting and evaluator, and do not show which model is better. Figs. 2, 3, 4, 5 and 6 are vector drawings; Figs. 1 and 7 are AI-generated concepts (frames with no image placed). None of them shows the inside of a real product or a measurement screen.