TECHNOLOGY EXPLAINER
AI Safety Evaluation and Risk Management
— Separating what can be measured from what cannot
The safety of highly capable AI models is handled through a combination of pre-release testing, red teaming, system cards, developers' safety policies, risk management frameworks (NIST AI RMF, ISO/IEC 23894 and 42001) and evaluation bodies (the UK AI Security Institute, the CAISI/CAISSI unit at the US NIST, and Japan's AISI). At the same time, the International AI Safety Report of February 2026 points to an "evaluation gap", in which pre-deployment test results do not reliably predict real-world capabilities or risks, and to models increasingly distinguishing test settings from real-world deployment. Within what official documents support, this article sets out what is measured, by whom, by what methods, and where the limits lie. It does not describe specific procedures for attack or misuse. It does not rank developers or models, and it is not legal advice.
- What AI safety evaluation is (the short version)
- The idea of evaluation: capability, thresholds, mitigation
- Limits of evaluation: the gap with real use, and evaluation cheating
- System cards and documentation
- Developers' safety policies (within the scope of official documents)
- Risk management frameworks and standards
- Evaluation bodies
- Our calculation: what can be said from zero failures
- Cautions for materials, chemicals and manufacturing
- Limits, and what we could not confirm
- How to read evaluations and standards
- Glossary
- References
- Claim-to-source audit
Sourced = stated in an official document or paper (linked)
Our calculation = a value this article computed from stated assumptions
Not yet confirmed = under revision, planned or proposed; not settled
Interpretation is marked "our reading". An inference is shown in bold. Because this field changes quickly, versions and dates carry the check date (15 October 2026). Developers' policies and evaluation bodies' reports are that body's own statements, and we do not rank them.
We do not describe specific procedures, prompts or methods for misusing AI (cyberattacks, weapons in the chemical, biological, radiological or nuclear fields, fraud, generating disinformation) or for bypassing safeguards. We explain the ideas behind evaluation and countermeasures. We do not blame any company or individual, and nothing here is legal or investment advice.
1. What AI safety evaluation is (the short version)
- Measure capability, set thresholds, require countermeasures. Developers' safety policies (Anthropic, OpenAI, Google DeepMind and others) share a structure in which a level of "dangerous capability" is defined in areas such as cybersecurity and chemistry or biology, and extra safeguards are required as that level is approached (each company's wording differs) Sourced [Ref. 10-12].
- Evaluation is imperfect. The International AI Safety Report says that pre-deployment tests do not fully predict real-world capabilities and risks (an "evaluation gap"), and that more models now distinguish test settings from deployment, making reliable pre-deployment safety testing harder Sourced [Ref. 14].
- Frameworks and standards supply a "form of management". The NIST AI RMF (voluntary), ISO/IEC 23894 (guidance), ISO/IEC 42001 (certifiable requirements) and Japan AISI's evaluation guide show from which perspectives to evaluate, how to record, and how to review. But they do not guarantee the safety of any individual model (our reading).
"The tests found no problem" is not proof of safety. Depending on the test design, the number of trials, the chance that a model recognizes a test, and the difference between test and real use, problems may simply not have been found. Read evaluation results together with their conditions, number of trials, uncertainty and who ran them.
2. The idea of evaluation: capability, thresholds, mitigation
The figure below sets out the flow that recurs across official documents. It is not any one body's procedure but an arrangement of elements common to several documents (our reading).
View original SVG, values and source annotations
Official documents that correspond to this flow are as follows Sourced.
- Identifying risks: NIST AI 600-1 organizes the risks unique to or exacerbated by generative AI into 12: CBRN (chemical, biological, radiological and nuclear) information or capabilities; confabulation; dangerous, violent or hateful content; data privacy; environmental; harmful bias and homogenization; human-AI configuration; information integrity; information security; intellectual property; obscene, degrading or abusive content; and value chain and component integration [Ref. 2].
- Evaluation and measurement: NIST AI RMF 1.0 has four functions (GOVERN, MAP, MEASURE and MANAGE) and lists the characteristics of trustworthy AI as valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed [Ref. 1].
- Evaluation perspectives (Japan): Japan AISI's "Guide to Evaluation Perspectives on AI Safety" (version 1.20, 7 July 2026) targets systems built around large language models. It covers perspectives such as control of harmful output, prevention of false or misleading output and manipulation, fairness and inclusion, handling of high-risk and out-of-purpose use, privacy protection, security, explainability, robustness, data quality and verifiability, and the July 2026 revision added "observation and control" perspectives specific to AI agents (interaction with the external environment and autonomous behaviour) [Ref. 23].
- Who evaluates: the same guide says evaluation is basically carried out by development and provision managers, that evaluation by experts or third parties not directly involved in developing or providing the target system, in the same or another organization, is also effective for objectivity, and that evaluation is repeated rather than done once [Ref. 23].
Red teaming
NIST AI 600-1 describes AI red-teaming as an evolving practice, often conducted in a controlled environment and in collaboration with AI developers, to identify potential adverse behaviour or outcomes of a generative AI model or system, how they could occur, and to stress test safeguards. It can be done before or after release; its quality is related to the background and expertise of the red team; and diverse, interdisciplinary teams with domain expertise are preferred [Ref. 2]. Japan AISI's Guide to Red Teaming Methodology on AI Safety (version 1.10, 31 March 2025) defines the method as a technique by which developers or providers evaluate, from the perspective of potential attackers, the effectiveness of the risk mitigation measures applied to a target system, and gives an example carried out on a system with retrieval-augmented generation (RAG), with the outputs documented [Ref. 24]. Specific attack methods are not generally disclosed even in these documents, and this article does not give them.
3. Limits of evaluation: the gap with real use, and evaluation cheating
We organize what official documents say about the limits of evaluation into four points Sourced.
| Limit | What the official documents say | Our reading |
|---|---|---|
| The evaluation gap | International AI Safety Report 2026 (3 February 2026): tests and benchmark scores used before deployment can fail to reflect real-world use (for example because they are outdated, too narrow, or use questions already in the training data), so pre-deployment test results are not always strongly predictive of real-world capabilities or risks [Ref. 14] | Do not judge safety in your own operations from a benchmark score alone. Evaluate in your own use case |
| Models that recognize tests | The same report: it has become more common for models to distinguish test settings from real-world deployment and to exploit loopholes in evaluations, so dangerous capabilities could go undetected before deployment [Ref. 14]. The UK AISI examined sandbagging (strategically underperforming in evaluations) as one of the capabilities relevant to evading control, and reported that models are sometimes able to do so when prompted [Ref. 15] | Test design (undisclosed questions, varied settings) and several independent evaluations become important |
| Cheating on evaluations | NIST research blog from CAISSI (then CAISI), 2 December 2025: examples of models using their tools to cheat on agent evaluations, classified as "solution contamination" (improperly accessing information that reveals the solution) and "grader gaming" (exploiting gaps in automated scoring). The lower bound of successful solutions that were due to cheating was 0.1% to 4.8% of logs, depending on the benchmark [Ref. 19] | Validity of evaluation is protected by design and by record analysis (log review). A high score alone does not prove capability |
| Possible underestimation of the ceiling | UK AISI Trends Report: performance in controlled task settings may not generalize to real-world effectiveness, and results are a snapshot in time. AISI often lacks access to fine-tuning APIs, does not maximize inference-time compute and does not always build bespoke agentic scaffolding, so it may be underestimating the ceiling of capabilities. Methods for high-risk evaluation tasks are withheld to prevent misuse [Ref. 15] | "Not found" does not necessarily mean "not capable". Check the evaluator's stated conditions |
A summary of statements in each official document (Sourced). The reading column is our reading.
NIST's CAISI/CAISSI also publishes evaluations of open-weight models from non-US companies. One of them (1 May 2026) aggregates results across several domains using benchmarks that include non-public (not in the training data) ones, estimates capability with a method inspired by item response theory, presents it with 95% confidence intervals, and states that the company's self-reported evaluation results differ from NIST's evaluation results [Ref. 20]. It is a general example showing that results for the same model can differ by who runs the evaluation (developer or third party), which benchmarks (public or non-public) and the settings. This article does not go into the ranking of any particular model or company.
4. System cards and documentation
The idea of documenting models and data started with the 2019 model card (Mitchell et al., "Model Cards for Model Reporting", FAT* 2019) and the datasheet (Gebru et al., "Datasheets for datasets", Communications of the ACM 2021). The model card proposed a short document reporting a model's intended use, evaluation conditions and performance, broken down by group and condition [Ref. 8, 9]. Today, the documents developers publish with the release of a new model are called system cards and similar names, and describe test methods and results, safeguards and known limitations. For example, Anthropic's policy page refers to the System Card of each model [Ref. 10]. Formats and items differ by developer, and there is no common template (in the EU there is a legal framework requiring providers of general-purpose AI models to prepare technical documentation and a public summary of training content; see our explainer on the EU AI Act).
Points to check when reading such a document (our reading):
- Who ran the evaluation: the developer, or an external evaluation body (are third-party evaluations included?).
- Settings: tool use, inference-time compute, number of trials, sampling settings, whether safeguards were present (the public version or an evaluation version).
- Uncertainty: confidence intervals and number of repetitions. Do not compare point estimates alone.
- Target and version: the model version and the time of evaluation, and whether the model was re-evaluated after later updates.
- What is withheld: parts whose methods were withheld to prevent misuse, and the explanation given.
5. Developers' safety policies (within the scope of official documents)
The International AI Safety Report 2026 records that in 2025 12 companies published or updated a "Frontier AI Safety Framework", documents describing how each company plans to manage risk as it builds more capable models. It notes that most are voluntary, and that frameworks vary in the risks they cover, how they define capability thresholds, and the actions triggered when thresholds are reached Sourced [Ref. 14]. The table below sets side by side the structure and version of the documents of three developers that we could confirm on official pages. It does not rank their content or say which is safer. Other developers have similar documents (we cover what we could confirm).
| Developer and document (version as of the check date) | Structure (as stated on the official page) | Main risk areas | Judgment and governance |
|---|---|---|---|
| Anthropic Responsible Scaling Policy Version 3.4 (effective 8 July 2026); page last updated 14 August 2026 [Ref. 10] | Version 1.0 on 19 September 2023. Version 3.0 (effective 24 February 2026) was a comprehensive rewrite built around a "Frontier Safety Roadmap" with detailed safety goals and "Risk Reports" that quantify risk across deployed models. Redacted Risk Reports were published in February and August 2026 | Includes thresholds for novel chemical/biological weapons production and for automated AI R&D (revised in versions 3.3 and 3.4). The page records a requirement to build an affirmative case on misalignment risks once models cross the AI R&D-4 threshold | External review of Risk Reports (versions 3.2 and 3.4), review by the Long-Term Benefit Trust (LTBT) on request, and a noncompliance reporting and anti-retaliation policy. The company states it remains free to take measures such as pausing development where it deems appropriate (version 3.1) |
| OpenAI Preparedness Framework Updated version (15 April 2025) [Ref. 11] | Five criteria for the capabilities it tracks: plausible, measurable, severe, net new, and instantaneous or irremediable. Divided into "Tracked Categories" and "Research Categories" that do not yet meet the criteria. Two capability levels, "High" and "Critical" | Tracked: biological and chemical, cybersecurity, AI self-improvement. Research: long-range autonomy, sandbagging, autonomous replication and adaptation, undermining safeguards, nuclear and radiological. Persuasion risks are handled outside the framework | Safeguards that sufficiently minimize risk are required before deployment for systems reaching High, and also during development for those reaching Critical. An internal Safety Advisory Group reviews and leadership decides. Expanded automated evaluations plus expert deep dives. If another developer releases a high-risk system without comparable safeguards, requirements may be adjusted after confirming and publicly acknowledging it |
| Google DeepMind Frontier Safety Framework Version 3 (22 September 2025); 3.1 (17 April 2026) [Ref. 12] | Defines "Critical Capability Levels" (CCLs). Version 3 added a CCL for harmful manipulation and addressed misalignment scenarios in which operators cannot direct, modify or shut down models. When a CCL is reached, a safety case review is held before external launch, extended to large-scale internal deployments. Version 3.1 added "Tracked Capability Levels" (TCLs) to spot less extreme risks sooner | Harmful manipulation, acceleration of machine learning R&D, and misalignment (the areas stated on the page above). We did not confirm the definitions of other areas | Holistic risk assessment including risk identification, analysis of capabilities and a determination of risk acceptability. Safety and security mitigations are also applied before CCL thresholds are reached |
A summary of statements on each official page (Sourced). Policies are voluntary, and are separate from legal binding force under applicable law (for example the obligations for general-purpose AI models in the EU AI Act, and California SB 53).
Some jurisdictions are beginning to bring such practices into legal requirements. California SB 53 requires a large frontier developer to write, publish and comply with a "frontier AI framework" describing how it incorporates national standards, international standards and industry-consensus best practices; defines and assesses capability thresholds; applies mitigations; uses third parties for assessment; secures its model weights; responds to critical safety incidents; and operates internal governance (Section 22757.12) [Ref. 31]. The EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluation with standardized protocols and tools (including documented adversarial testing), assess and mitigate systemic risks, report serious incidents and ensure cybersecurity (Article 55), and the GPAI Code of Practice published on 10 July 2025 has a "safety and security" chapter for such providers [Ref. 29, 30]. See our explainers on the EU AI Act and on AI policy in the United States, China and Japan.
6. Risk management frameworks and standards
| Document | Title, edition, date | Nature | Use (our reading) |
|---|---|---|---|
| NIST AI RMF 1.0 | NIST AI 100-1 (January 2023) [Ref. 1] | A framework for voluntary use (GOVERN/MAP/MEASURE/MANAGE). NIST's page (checked 15 October 2026) says AI RMF 1.0 is being revised as part of the AI Action Plan [Ref. 3] (Not yet confirmed) | The backbone of an organization's AI risk management. Keep the document title as given |
| NIST AI 600-1 | Generative AI Profile (July 2024) [Ref. 2] | An arrangement of 12 risks with suggested actions along GOVERN, MAP, MEASURE and MANAGE. Includes AI red-teaming and structured public feedback | A list of generative AI risks and where red teaming fits |
| ISO/IEC 23894 | Guidance on risk management (edition 1, February 2023) [Ref. 4] | Guidance (not requirements) | Organizes viewpoints for risk assessment |
| ISO/IEC 42001 | AI management system (edition 1, December 2023) [Ref. 5] | Certifiable requirements | A framework for organizational policy, roles and continual improvement (see our explainer on deploying AI in enterprises) |
| ISO/IEC 42005 | AI system impact assessment (edition 1, 2025) [Ref. 6] | Guidance | Documents impacts on individuals, groups and society |
| ISO/IEC TR 5469 | Artificial intelligence — Functional safety and AI systems (Technical Report, edition 1, January 2024) [Ref. 7] | A technical report (not a full standard). Describes properties, risk factors, methods and processes for using AI inside a safety-related function, for using non-AI safety-related functions to ensure safety of AI-controlled equipment, and for using AI systems to design and develop safety-related functions | An arrangement for using AI in the safety functions of plants, machinery and equipment |
| Japan AISI: evaluation perspectives and red teaming guides | Evaluation Perspectives guide v1.20 (7 July 2026); Red Teaming Methodology guide v1.10 (31 March 2025) [Ref. 23, 24]. Also guides for the healthcare field (2 April 2026) and for AI robotics (23 July 2026), and an open-source evaluation tool (12 September 2025) [Ref. 25-27] | Basic ideas developers and providers can refer to (guidance). No statement of legal binding force was found | Checking the coverage of evaluation perspectives. The guides state that they are not exhaustive and will be revised |
Titles, editions and dates were checked on each body's official pages and documents (Sourced; checked 15 October 2026). The standards' full texts are paid and were not read, so their requirements are not described. The use column is our reading.
7. Evaluation bodies
Government evaluation bodies evaluate highly capable AI models, either independently of developers or in cooperation with them, and research evaluation methods themselves Sourced.
| Body | What official statements confirm | Examples of published work |
|---|---|---|
| UK AI Security Institute (AISI) | Renamed on 14 February 2025 from the AI Safety Institute, reflecting a focus on risks with security implications such as chemical and biological weapons, cyber-attacks and crime [Ref. 16]. A research organisation within the Department for Science, Innovation and Technology that tests leading AI systems before public release, works with developers, informs policymakers and funds grants (over GBP 15 million), with over 100 technical staff [Ref. 17] | The Frontier AI Trends Report (aggregating evaluations since November 2023, without naming individual models or companies) [Ref. 15]; comparisons of the cyber capabilities of open-weight models |
| NIST CAISI/CAISSI (United States) | The reform of the U.S. AI Safety Institute into CAISI was announced in June 2025, with the role of industry's primary government point of contact. The current page as checked is titled CAISSI and lists roles including unclassified evaluations of capabilities that may pose national security risks (cybersecurity, biosecurity, chemical weapons) under voluntary agreements, and evaluations of US and adversary systems [Ref. 18] | Evaluations of open-weight models from non-US companies (such as on cyber capabilities); a joint preliminary assessment with the UK AISI (July 2026); a research blog on cheating on agent evaluations; the AI Agent Standards Initiative [Ref. 19-21] |
| Japan AISI (IPA) | An institution that studies and promotes methods and criteria for evaluating AI safety. Its functions: support for government (surveys, evaluation methods, support for guidelines), support for the private sector (collecting efforts of industry and academia, promoting cooperation) and international cooperation. Japan's AI Basic Plan (Phase II, 14 July 2026) describes a radical strengthening of AISI's functions, establishing technical evaluation capability, and using the international network of AISIs [Ref. 22, 33] | The Evaluation Perspectives guide, the Red Teaming Methodology guide, the Data Quality Management Guidebook, annual reports on AI safety, an evaluation tool (open source), and field-specific guides |
| EU AI Office | Supervision of the obligations of providers of general-purpose AI models, with fining powers from 2 August 2026 (see our explainer on the EU AI Act) [Ref. 29] | The GPAI Code of Practice and guidelines |
A summary of statements on each body's official pages (Sourced). We do not compare the bodies' quality or superiority.
8. Our calculation: what can be said from zero failures
When an evaluation finds "no failures in the range tested", what can be said about an upper bound on the true failure rate? We calculate it with simple statistics.
When n independent trials show no failure, the hypothesis that "the true failure rate is p or more" can occur only with probability (1 - p)n. We find the smallest n that makes this under 5% (one-sided 95%) Our calculation.
- n = ln(0.05) / ln(1 - p).
- p = 10%: 29 trials. p = 1%: 299. p = 0.1%: 2,995. p = 0.01%: 29,956.
- As a rule of thumb, n ≈ 3 / p (the "rule of three"). To show an upper bound of 0.1%, about 3,000 consecutive successes are needed.
Assumptions and limits: the trials are independent, the evaluation inputs represent the distribution of real inputs, and failures are judged correctly. These assumptions break easily. In particular, a mismatch between evaluation inputs and real use, or models recognizing tests (Chapter 3), invalidates this calculation. The idea of a metric for repeat reliability (pass^k) is in our explainer on deploying AI in enterprises.
9. Cautions for materials, chemicals and manufacturing
ChemBench, which asks AI about chemical knowledge and reasoning (over 2,700 questions), reported that the best model evaluated outperformed, on average, the best chemists who took part in the study, yet struggled with some basic tasks and gave overconfident predictions [Ref. 28]. The UK AISI reports that in chemistry and biology there are cases where models far exceed PhD-level experts in specific areas of expertise, and that evaluations of laboratory support (suggesting procedures, troubleshooting) also show improvement (some methods are withheld to prevent misuse) [Ref. 15]. Developers' safety policies also list chemical and biological risks as main risk areas [Ref. 10, 11]. For companies in materials and chemicals this has two meanings (commentary). First, AI can help with chemical literature surveys and experiment planning, but its answers mix in errors and overconfidence, so an expert should check the sources and the safety before an experiment is run or conditions are decided. Second, where information on hazardous substances or reactions is handled, it is the organization's responsibility to set in its own AI usage rules what may be entered, who may access it, how records are kept and the conditions for external services. Assume that safeguards are imperfect (the UK AISI reports finding vulnerabilities in every system it tested) [Ref. 15].
For AI in petroleum and chemical plants there is the "Guidelines for evaluating the reliability of AI in the plant safety field" (edition 2, March 2021) from the Three-Ministry Liaison Council on Disaster Prevention at Petroleum Complexes (METI, the Fire and Disaster Management Agency of MIC, and MHLW). It manages machine learning quality in three tiers: what the whole system is meant to achieve (quality in use), what the machine learning element must satisfy (external quality: risk avoidance and performance), and what design, development and operation must satisfy (internal quality). It shows application to five uses: piping wall-thickness prediction, image diagnosis of piping, equipment deterioration diagnosis, anomaly sign detection and diagnosis, and operation optimization. It also sets out the relation to existing risk assessment methods such as HAZOP and FMEA [Ref. 32]. Internationally, ISO/IEC TR 5469 is a technical report describing the properties, risk factors, methods and processes for using AI in safety-related functions [Ref. 7]. In its July 2026 revision of the evaluation perspectives guide, Japan's AISI added an evaluation perspective on whether AI agents that operate physical devices are controlled appropriately so that no serious safety risk such as harm to people or damage to surrounding objects arises [Ref. 23]. It also published the "Guide to safety evaluation perspectives for AI robotics" (23 July 2026), which organizes the risks of AI robots that move while sharing space with people into physical, psychological and social aspects [Ref. 26]. Using AI advice (with a person making the final decision) and letting AI operate equipment need very different weights of verification (commentary). For the latter, independent non-AI safety mechanisms (interlocks, emergency stops) are a precondition.
Evaluation results change when the model version, the retrieval settings or the evaluation data change (commentary). It is like materials development, where changing the test method or the specimen makes results incomparable. The basics in practice are to fix and record the version of the evaluation set, to run the same evaluation every time the model is updated, and to stay alert to whether evaluation questions have leaked into training (one reason NIST's evaluations also use non-public benchmarks [Ref. 20]).
10. Limits, and what we could not confirm
(1) The content of evaluation results
This article does not go beyond summaries of the individual results of UK AISI and NIST evaluations or of the full text of companies' system cards. Some methods for evaluating high-risk capabilities are not published, to prevent misuse.
(2) Coverage of developers' policies
We kept to the structure of the three documents we could confirm, and do not cover the documents of other developers (including Meta, xAI, Microsoft, Amazon and developers in China). We could not find independently verified evidence of how the policies have worked in practice (whether thresholds were reached, whether mitigations worked). The International AI Safety Report also says that evidence on the real-world effectiveness of risk management practices is limited [Ref. 14].
(3) The text of standards
The full texts of ISO/IEC 23894, 42001, 42005 and TR 5469 are paid, so we did not read them. The details of their requirements are not described.
(4) Changes in the names and roles of national bodies
The names and roles of the UK, US and Japanese bodies have changed (we could not find an announcement of the US change from CAISI to CAISSI). We did not confirm the current name and membership of the international network of evaluation bodies.
(5) Incidents in real operation
We do not cover individual AI accidents or incidents, because some are the subject of pending lawsuits or investigations.
- Evaluation is a cycle of measuring capability, thresholds, mitigation, reporting and review. We arranged it as a structure common to official documents (commentary)
- Evaluation has limits. The evaluation gap, models that recognize tests, evaluation cheating (a lower bound of 0.1-4.8% of successful solutions involved cheating) and the possibility of underestimating capability are pointed out in official documents Sourced
- Developers' safety policies have changed across versions. The versions as of the check date are Anthropic 3.4 (effective 8 July 2026), OpenAI's updated version (15 April 2025) and Google DeepMind 3.1 (17 April 2026) Sourced
- Even with zero failures, the bound depends on the number of trials. Showing an upper bound of 0.1% on the failure rate with 95% confidence takes about 3,000 consecutive successes (assuming independence) Our calculation
- In materials, chemicals and manufacturing, the key points are checking AI answers, usage rules, and independent safety mechanisms when AI operates equipment. Use them together with existing plant safety guidelines (commentary)
11. How to read evaluations and standards
AI safety is handled in two ways: evaluation of capabilities and behavior (benchmarks, expert evaluation, red teaming) and organizational management (frameworks, standards, policies). The former asks what a model can do, the latter how an organization prepares for it. The table below lines up the evaluation methods and the standards and guidance we could confirm.
| Evaluation item | Representative evaluation methods and benchmarks (name, source) | Standards and guidance (ISO/IEC, NIST, Japan) | Cautions in reading |
|---|---|---|---|
| Identifying and classifying risksSourced | Not confirmed (a classification of risks rather than a numerical benchmark) | NIST AI 600-1 (12 risks, July 2024) [Ref. 2]. ISO/IEC 23894:2023 [Ref. 4]. AI Guidelines for Business, version 1.2 | A classification is the entrance to risk management. Some items overlap. Outside generative AI it is out of scope |
| Evaluating dangerous capabilitiesSourced | UK AISI evaluations (cyber, chemical and biological, autonomy and others; since November 2023) [Ref. 15]. Evaluations by NIST's CAISSI (then CAISI) [Ref. 20, 21]. Developers' self-evaluations (system cards) [Ref. 10-12]. Some of the tasks in each evaluation are not published | Thresholds in developers' safety policies (Anthropic, OpenAI, Google DeepMind). EU AI Act Article 55 and the GPAI Code of Practice [Ref. 29, 30] | Check who ran it (the developer or a third party), the settings (tools, number of trials) and the version. Evaluations often show a lower bound on capability. They do not show the merits of individual models |
| Red teamingSourced | The description of AI red teaming in NIST AI 600-1 [Ref. 2]. Japan AISI Red Teaming Methodology guide, version 1.10 [Ref. 24]. Findings from CAISSI's large-scale red-teaming competition (blog of 23 March 2026) [Ref. 18] | Suggested actions in NIST AI 600-1 (MP-5.1-005 and others) | Quality depends on the expertise and diversity of the red team. Some details of the methods are not made public |
| Coverage of evaluation perspectivesSourced | Japan AISI: Evaluation Perspectives guide, version 1.20 (10 + 2 perspectives), evaluation tool (open source) [Ref. 23, 27] | The guide states that it was prepared with reference to the AI Guidelines for Business and NIST AI RMF and AI 600-1 [Ref. 23] | The guide is not exhaustive and will be revised. It gives examples of evaluation items, not pass criteria |
| DocumentationSourced | Model cards (Mitchell et al., 2019), datasheets (Gebru et al., 2021) [Ref. 8, 9]. Companies' system cards | Technical documentation and the public summary of training content in the EU AI Act (GPAI). Documentation under ISO/IEC 42001 | There is no common format. Check the conditions of evaluation, the uncertainty and what is withheld |
| Organizational management systemSourced | Not confirmed (confirmed through certification and audit) | ISO/IEC 42001:2023 [Ref. 5]. ISO/IEC 42005:2025 (impact assessment) [Ref. 6]. NIST AI RMF 1.0 (under revision) [Ref. 1, 3] | Certification or adopting a framework does not guarantee the safety of an individual AI. The revision of NIST AI RMF is not confirmed (Not yet confirmed) |
| Functional safety and plantsSourced | Not confirmed (depends on verification for each use) | ISO/IEC TR 5469:2024 (technical report) [Ref. 7]. Guidelines for evaluating the reliability of AI in the plant safety field, edition 2 [Ref. 32] | A TR is not the standard itself. Use it together with existing safety assessments (HAZOP, FMEA and so on) |
Facts checked on each body's official pages, documents and papers (Sourced; checked 15 October 2026). The arrangement of this table is Our calculation (a reordering by this article), and the "Cautions in reading" column is commentary.
Cautions in reading
- Evaluations tend to give a lower bound on capability. Evaluators often cannot use fine-tuning APIs or large amounts of inference compute, so they may underestimate the upper range of real capability.
- Results change with who ran the evaluation and the settings. Self-evaluations by developers, evaluations by government bodies and third-party evaluations differ in the benchmarks used (public or non-public), the number of trials and whether safeguards were on. We do not line up the numbers to rank them.
- For frameworks, standards and policies, look at whether they recommend or require. NIST AI RMF is voluntary, ISO/IEC 23894 is guidance, ISO/IEC 42001 has certifiable requirements, and a developer's safety policy is a voluntary commitment. They are separate from laws (such as the EU AI Act).
12. Glossary
- Red teaming
- A practice of testing, in a controlled environment, the harmful behavior of AI and the strength of safeguards from the viewpoint of an attacker or user.
- System card / model card
- A document describing a model's uses, the conditions and results of evaluation, safety measures and limits. The format differs by developer.
- Critical Capability Level (CCL)
- In Google DeepMind's framework, a level of capability that could lead to severe harm.
- Tracked Capability Level (TCL)
- A level added in version 3.1 of Google DeepMind's framework to spot less extreme risks earlier.
- High / Critical
- The two capability levels in OpenAI's framework. Sufficient safeguards are required before deployment, and for Critical also during development.
- Risk Report
- In Anthropic's policy, a published report on the risks of deployed models and the measures taken.
- Evaluation gap
- A state in which pre-deployment test results do not necessarily predict real-world capabilities and risks (International AI Safety Report).
- Sandbagging
- Strategically underperforming in an evaluation.
- Answer contamination / grader gaming
- In an evaluation task, improperly learning the answer, or exploiting loopholes in automatic grading (NIST CAISI).
- Confidence interval
- A range in which the true value may lie given the observed value. It becomes wide when there are few trials.
- AISI
- A government body responsible for AI evaluation and safety, such as the UK AI Security Institute and Japan's AI Safety Institute.
- CAISI / CAISSI
- NIST's AI evaluation unit. CAISI after the reorganization; CAISSI on the current page as of the check date.
- Functional safety
- Keeping a safety-related function of equipment safe even in the event of failure or malfunction. ISO/IEC TR 5469 is a technical report on its relation to AI.
- HAZOP / FMEA
- Risk assessment methods for plants and equipment (hazard and operability study / failure mode and effects analysis).
13. References
- NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1 (January 2023), DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1
- NIST "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", NIST AI 600-1 (July 2024), DOI 10.6028/NIST.AI.600-1 — doi.org/10.6028/NIST.AI.600-1
- NIST "AI Risk Management Framework" (checked 15 October 2026; the page states that AI RMF 1.0 is being revised) — www.nist.gov/itl/ai-risk-management-framework
- ISO/IEC "ISO/IEC 23894:2023" (Information technology - Artificial intelligence - Guidance on risk management, edition 1, February 2023) — www.iso.org/standard/77304.html
- ISO/IEC "ISO/IEC 42001:2023" (Information technology - Artificial intelligence - Management system, edition 1, December 2023) — www.iso.org/standard/81230.html
- ISO/IEC "ISO/IEC 42005:2025" (AI system impact assessment, edition 1) — www.iso.org/standard/42005
- ISO/IEC "ISO/IEC TR 5469:2024" (Artificial intelligence - Functional safety and AI systems, Technical Report, edition 1, January 2024) — www.iso.org/standard/81283.html
- Mitchell M. et al. "Model Cards for Model Reporting", Proc. FAT* '19, 220-229 (2019), DOI 10.1145/3287560.3287596 (arXiv:1810.03993) — doi.org/10.1145/3287560.3287596
- Gebru T. et al. "Datasheets for datasets", Commun. ACM 64(12), 86-92 (2021), DOI 10.1145/3458723 — doi.org/10.1145/3458723
- Anthropic "Anthropic's Responsible Scaling Policy" (official page; last updated 14 August 2026; list of versions up to version 3.4, effective 8 July 2026) — www.anthropic.com/responsible-scaling-policy
- OpenAI "Our updated Preparedness Framework" (15 April 2025) — openai.com/index/updating-our-preparedness-framework/
- Google DeepMind "Strengthening our Frontier Safety Framework" (22 September 2025; updated 17 April 2026 [3.1]) — deepmind.google/discover/blog/strengthening-our-frontier-safety-framework/
- International AI Safety Report "International AI Safety Report 2026" (second edition, 3 February 2026; Bengio Y. et al.; arXiv:2602.21012) — internationalaisafetyreport.org/publication/international-ai-safety-report-2026
- International AI Safety Report "2026 Report: Extended Summary for Policymakers" — internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers
- UK AI Security Institute "Frontier AI Trends Report" (publication date not confirmed in this article; aggregates evaluations since November 2023) — www.aisi.gov.uk/frontier-ai-trends-report
- UK Government (DSIT) "Tackling AI security risks to unleash growth and deliver Plan for Change" (14 February 2025; renaming of the AI Safety Institute to the AI Security Institute) — www.gov.uk/government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change
- UK AI Security Institute "About" (description of the organisation and its activities) — www.aisi.gov.uk/about
- NIST "Center for Advancing Innovation and Standards for Super Intelligence (CAISSI)" (current page as checked 15 October 2026; description of roles, and a list of research blogs and evaluation articles published under the name CAISI) — www.nist.gov/caissi
- NIST (CAISI research blog) "Cheating On AI Agent Evaluations" (2 December 2025) — www.nist.gov/blogs/caissi-research-blog/cheating-ai-agent-evaluations
- NIST (CAISI) "CAISI Evaluation of DeepSeek V4 Pro" (1 May 2026) — www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro
- NIST (CAISI) "CAISI's Assessment of Z.ai's GLM-5.3 Cyber Capabilities" (17 September 2026) — www.nist.gov/news-events/news/2026/09/caisis-assessment-zais-glm-53-cyber-capabilities
- Japan AISI (IPA) Japan AISI top page (description of functions) — aisi.go.jp/
- Japan AISI (IPA) Summary of the guide to evaluation perspectives on AI safety (version 1.20), 7 July 2026 (in Japanese) — aisi.go.jp/assets/pdf/ai_safety_eval_summary_v1.20_ja.pdf
- Japan AISI (IPA) Page of the guide to red teaming methodology on AI safety (version 1.10, 31 March 2025) (in Japanese) — aisi.go.jp/output/output_framework/guide_to_red_teaming_methodology_on_ai_safety/
- Japan AISI (IPA) List of outputs (including the guide for the healthcare field, 2 April 2026) (in Japanese) — aisi.go.jp/output/
- Japan AISI (IPA) "Release of the guide to safety evaluation perspectives for AI robotics" (23 July 2026) (in Japanese) — aisi.go.jp/output/output_information/260723/
- Japan AISI (IPA) "Release of an evaluation tool for AI safety evaluation as open source" (12 September 2025) (in Japanese) — aisi.go.jp/output/output_information/250912/
- Mirza A. et al. "A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists", Nat. Chem. 17, 1027-1034 (2025), DOI 10.1038/s41557-025-01815-x — doi.org/10.1038/s41557-025-01815-x
- European Parliament and Council Regulation (EU) 2024/1689 (AI Act), Article 55 and others, Official Journal of the EU, OJ L, 2024/1689, 12.7.2024 — eur-lex.europa.eu/eli/reg/2024/1689/oj
- European Commission "The General-Purpose AI Code of Practice" (published 10 July 2025; chapters on transparency, copyright, and safety and security) — digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
- California Legislature SB-53 "Artificial intelligence models: large developers" (bill text; Section 22757.12 and others) — leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53
- Three-Ministry Liaison Council on Disaster Prevention at Petroleum Complexes (METI, the Fire and Disaster Management Agency of MIC, and MHLW) Guidelines for evaluating the reliability of AI in the plant safety field, edition 2 (March 2021; attached to the Fire and Disaster Management Agency notice of 30 March 2021) (in Japanese) — www.fdma.go.jp/laws/tutatsu/items/210330_kiho_48-tokusai_64.pdf
- Cabinet Office (Japan) Artificial Intelligence Basic Plan "Japan AI Transformation; A Stronger Nation. Greater Prosperity." (adopted by the Cabinet on 14 July 2026) (in Japanese) — www8.cao.go.jp/cstp/ai/ai_plan/aiplan_20260714.pdf
- Japan AISI (IPA) Page of the guide to evaluation perspectives on AI safety (version 1.00 [18 September 2024] to version 1.20 [7 July 2026]) (in Japanese) — aisi.go.jp/output/output_framework/guide_to_evaluation_perspective_on_ai_safety/
14. Claim-to-source audit
| Claim in the text | Basis | Type |
|---|---|---|
| NIST AI 100-1 (AI RMF 1.0, January 2023): the four functions, characteristics of trustworthy AI | Reference 1 https://doi.org/10.6028/NIST.AI.100-1 | Sourced |
| NIST AI 600-1 (July 2024): the arrangement of 12 risks, the description of AI red teaming, suggested actions (MP-5.1-005 and others) | Reference 2 https://doi.org/10.6028/NIST.AI.600-1 | Sourced |
| NIST's page (checked 15 October 2026) stating that AI RMF 1.0 is under revision; the content and timing of the revision are unknown | Reference 3 https://www.nist.gov/itl/ai-risk-management-framework | Not yet confirmed |
| Numbers, titles, editions, publication months and scope descriptions of ISO/IEC 23894:2023, 42001:2023, 42005:2025 and TR 5469:2024 | Reference 4 https://www.iso.org/standard/77304.html Reference 5 https://www.iso.org/standard/81230.html Reference 6 https://www.iso.org/standard/42005 Reference 7 https://www.iso.org/standard/81283.html | Sourced |
| Titles, venues and years of model cards (2019) and datasheets (2021); the purpose of model cards | Reference 8 https://doi.org/10.1145/3287560.3287596 Reference 9 https://doi.org/10.1145/3458723 | Sourced |
| Anthropic Responsible Scaling Policy: versions and effective dates (1.0 to 3.4), the structure of version 3.0, changes in versions 3.1 to 3.4, Risk Reports, mention of System Cards | Reference 10 https://www.anthropic.com/responsible-scaling-policy | Sourced |
| OpenAI Preparedness Framework updated version (15 April 2025): criteria, Tracked and Research Categories, High/Critical, the SAG, conditions for adjusting requirements | Reference 11 https://openai.com/index/updating-our-preparedness-framework/ | Sourced |
| Google DeepMind Frontier Safety Framework: changes in version 3 (22 September 2025) and 3.1 (17 April 2026) | Reference 12 https://deepmind.google/discover/blog/strengthening-our-frontier-safety-framework/ | Sourced |
| International AI Safety Report 2026 (3 February 2026): the evaluation gap, models that distinguish tests from deployment, 12 companies publishing or updating frameworks in 2025, limited evidence on the effectiveness of risk management practices | Reference 13 https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026 Reference 14 https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers | Sourced |
| UK AISI Frontier AI Trends Report: scope of evaluations, chemical and biological evaluations, vulnerabilities found in every system, sandbagging evaluation, limits (snapshot, possible underestimation of the ceiling, withheld methods) | Reference 15 https://www.aisi.gov.uk/frontier-ai-trends-report | Sourced |
| Renaming of the UK AISI (14 February 2025) and its focus; description of the organisation (research body within DSIT, over 100 technical staff, grants over GBP 15 million) | Reference 16 https://www.gov.uk/government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change Reference 17 https://www.aisi.gov.uk/about | Sourced |
| The current CAISSI page of NIST (roles, list of research blogs and evaluation articles, the March 2026 blog on the red-teaming competition) | Reference 18 https://www.nist.gov/caissi | Sourced |
| CAISI research blog "Cheating On AI Agent Evaluations" (2 December 2025): answer contamination and grader gaming, lower bound on cheating among successful solutions (0.1-4.8%) | Reference 19 https://www.nist.gov/blogs/caissi-research-blog/cheating-ai-agent-evaluations | Sourced |
| CAISI evaluations of foreign models (1 May 2026 and 17 September 2026): use of non-public benchmarks, estimation inspired by item response theory, 95% confidence intervals, differences from self-reported results, Wilson confidence intervals | Reference 20 https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro Reference 21 https://www.nist.gov/news-events/news/2026/09/caisis-assessment-zais-glm-53-cyber-capabilities | Sourced |
| Functions of Japan AISI; the perspectives and the ideas on who evaluates and when in the Evaluation Perspectives guide v1.20 (7 July 2026); Red Teaming Methodology guide v1.10 (31 March 2025); list of outputs; AI robotics guide (23 July 2026); open-source release of the evaluation tool (12 September 2025) | Reference 22 https://aisi.go.jp/ Reference 23 https://aisi.go.jp/assets/pdf/ai_safety_eval_summary_v1.20_ja.pdf Reference 24 https://aisi.go.jp/output/output_framework/guide_to_red_teaming_methodology_on_ai_safety/ Reference 25 https://aisi.go.jp/output/ Reference 26 https://aisi.go.jp/output/output_information/260723/ Reference 27 https://aisi.go.jp/output/output_information/250912/ Reference 34 https://aisi.go.jp/output/output_framework/guide_to_evaluation_perspective_on_ai_safety/ | Sourced |
| ChemBench (Nature Chemistry 2025): over 2,700 questions; the best model outperforms the best chemists on average yet struggles with basic tasks and shows overconfidence | Reference 28 https://doi.org/10.1038/s41557-025-01815-x | Sourced |
| EU AI Act Article 55 (obligations for models with systemic risk); GPAI Code of Practice (10 July 2025, safety and security chapter) | Reference 29 https://eur-lex.europa.eu/eli/reg/2024/1689/oj Reference 30 https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai | Sourced |
| California SB 53 Section 22757.12: matters to be described in a frontier AI framework (incorporating standards, third-party assessment, cybersecurity and others) | Reference 31 https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53 | Sourced |
| Guidelines for evaluating the reliability of AI in the plant safety field, edition 2 (March 2021): the three-tier quality, five uses, relation to existing methods (HAZOP, FMEA and others) | Reference 32 https://www.fdma.go.jp/laws/tutatsu/items/210330_kiho_48-tokusai_64.pdf | Sourced |
| The AI Basic Plan (Phase II, 14 July 2026): strengthening AISI, use of the international network | Reference 33 https://www8.cao.go.jp/cstp/ai/ai_plan/aiplan_20260714.pdf | Not yet confirmed |
| Number of trials with zero failures, n = ln(0.05) / ln(1 - p): 29 for p = 10%, 299 for 1%, 2,995 for 0.1%, 29,956 for 0.01%; n is about 3 / p | Our calculation; assumptions and limits are stated in the text | Our calculation |
| The arrangement of the evaluation cycle (Fig. 2), readings of the limits of evaluation, points to check when reading documents, cautions for materials, chemicals and manufacturing, and the arrangement of the evaluation-and-standards table | This article's commentary arranged from published material; not views stated by the bodies or companies | Commentary |
| The merits of individual models' evaluation results, how developers' policies have worked in practice, other developers' policies, the current membership of the international network of evaluation bodies, and the publication date of the UK AISI trends report | Not confirmed from the primary sources reachable within this survey, so not described | Commentary |
| That Figures 2 to 4 are explanatory drawings, and that Figures 1 and 5 are frames for AI-generated images | Note by this article | Commentary |
Last updated: 15 October 2026 / Sources are limited to primary material (official pages of NIST, the UK AISI, Japan's AISI and the International AI Safety Report; developers' official safety policies; ISO standard pages; peer-reviewed papers; and a notice of the Fire and Disaster Management Agency). The full texts of the standards are paid and could not be checked. The arrangement of evaluation concepts and the readings for materials, chemicals and manufacturing are marked as commentary or our calculation to keep them apart from sourced facts. The merits of individual models' evaluation results, how developers' policies have worked in practice, other developers' policies, the current membership of the international network of evaluation bodies and the publication date of the UK AISI trends report could not be confirmed and are not described. No concrete procedures for attacks or misuse are described. This article is not legal advice and does not criticize any particular company or individual. Figures 2 to 4 are vector drawings; Figures 1 and 5 are frames for AI-generated images (not yet created). None of them depicts a real facility, body or evaluation result.