1. The short version / before you start / where this fits in the series

• Before reading a number, check four questions: is the correct answer (label) measured or calculated, how was the test data split, does the metric fit the use, and does the same result come out again? With these four, the same "error of 0.1 eV" can be a useful number or an almost meaningless one.
• Small errors, yet wrong decisions: there is a report that machine learning predicts formation energy well but predicts stability poorly Sourced [Reference 5]. In synthetic data, even with a mean absolute error of 0.06 eV/atom, only 72.5% of the candidates called "stable" were truly stable Our calculation.
• "AI discovered" comes in levels, and there is criticism and a correction: for the autonomous laboratory A-Lab, an author correction of 19 January 2026 clarified "novel" as "new to the prediction platform" and the numbers in the abstract were updated Sourced [References 10, 11]. Another group examined the report in detail and argues that the discovery of new materials is not supported [Reference 13]. What matters is reading how far the claim goes.

2. Four places where MI evaluation goes wrong

Pitfalls of evaluation in machine learning in general are covered in other articles. Once the circumstances specific to materials are added, the places where things go wrong can be organized into the following four (our own organization).

• 1. What is the label: most materials data are produced by calculation such as DFT. The accuracy of matching calculated values differs from the accuracy of matching measurements (we explained this with band gaps in chapter 3 of our explainer on applications to semiconductor and electronic materials).
• 2. How was it split: there are many similar materials (the same chemical system, the same family of crystal structures), so if the data are split at random, the "siblings" of each test record remain on the training side and the numbers become optimistic (chapter 5).
• 3. Does the metric fit the purpose: the purpose of a search is a decision: "is it stable?", "hit or miss?". Even with a small error (MAE), decisions near the boundary go wrong (chapter 4).
• 4. Does the same result come out again: calculations change with code and settings. Experiments change with laboratory, instrument and sample. We need to decide what counts as "the same" (chapter 6).

3. Shared yardsticks: Matbench, Matbench Discovery, JARVIS-Leaderboard

In materials MI there are shared benchmarks for comparing methods under the same conditions. We summarize the official statements of three representative ones (checked 22 October 2026). All three are community efforts, not standards.

(1) Matbench: comparing property prediction

Matbench is a shared set of problems for predicting properties of inorganic bulk materials, released by Dunn and co-workers in 2020. It has 13 tasks (from 312 to 132,752 records, from 10 data sources), predicting optical, thermal, electronic, thermodynamic, tensile and elastic properties from composition or crystal structure. All tasks are evaluated with the same 5-fold nested cross-validation Sourced [Reference 1]. The authors say that this "measure every task by one procedure" approach is easy for everyone to use, but that cross-validation split by group may be closer to real use (predicting chemical systems with no data) in some cases [Reference 1].

(2) Matbench Discovery: comparing the judgement of crystal stability

Matbench Discovery by Riebesell and co-workers is a framework that compares machine-learning energy models, used as a pre-filter ahead of first-principles calculation, on the task of searching for stable inorganic crystals Sourced [Reference 2]. The paper lists four points as design ideas.

• A realistic task (prospective data): evaluate on newly generated data from the intended search workflow. The test data are the WBM dataset (257,487 structures), made by element substitution and DFT relaxation.
• Relevant targets: use the "distance to the convex hull", an indicator of stability, rather than formation energy.
• Informative metrics: MAE, RMSE and R² alone can give misleading confidence in reliability. Look mainly at performance as decisions (classification metrics).
• Scalability: look at behaviour on large data.

The paper's evaluation uses the "discovery acceleration factor (DAF)". The DAF is the share of stable materials among the candidates picked by a model (precision) divided by the share of stable materials when picking at random (prevalence); its maximum on the test data is about 6.5 (about 33,000 stable out of 215,000 structures) [Reference 2]. The same paper points out that even accurate regression models have unexpectedly many false positives (actually unstable but called stable) when the prediction is close to the stability boundary (0 eV/atom above the hull) [Reference 2].

The official website's leaderboard (checked 22 October 2026) lists, besides discovery (judgement), tasks such as geometry optimization, phonons and molecular dynamics, and can be sorted by a combined score CPS (discovery F1 at 50%, structure RMSD at 10% and the phonon thermal-conductivity error κ SRME at 40%) Sourced [Reference 3]. The display can be switched between "full test set" and "unique prototypes only". The weights are a design choice of the site, and what to emphasize is for the user to decide (commentary). The leaderboard and the weights of the metrics may keep changing, so read the numbers as of the checking date Not yet confirmed.

(3) JARVIS-Leaderboard: comparing AI, calculation and experiment in one place

JARVIS-Leaderboard by Choudhary and co-workers is an open-source, community-run platform for comparison, built to address the lack of reproducibility checks and validation. It covers five areas: AI, electronic-structure calculations (ES), force fields (FF), quantum computation (QC) and experiments (EXP); AI takes atomic structures, microscope images, spectra and text as input. For electronic-structure calculations it varies methods, software, pseudopotentials, materials and properties and compares with experiment, and for experiments it builds benchmarks by an interlaboratory approach (several laboratories measuring the same thing). At the time of the paper there were 1,281 contributions to 274 benchmarks (152 methods, more than 8 million data points) Sourced [Reference 4].

4. Small errors, wrong decisions: judging stability

(1) Formation energy is predicted well, stability is not

When searching for candidates for new materials, what we want to know is "can this compound exist stably?". Stability is set by competition with other phases of the same elements, and we examine "is it better to decompose into other phases?" with a diagram called the convex hull. If the distance to the hull (decomposition enthalpy ΔHd) is 0 or less the compound is judged stable, and if it is above 0, unstable.

In studies that predict the formation energy of compounds by machine learning (the energy ΔHf of making a compound from its elements), there were claims of accuracy approaching DFT. Bartel and co-workers tested the formation energies of seven machine-learning models for stability prediction on 85,014 compositions of the Materials Project. The result was that formation energy itself is predicted well, but every model that predicts from composition alone is poor at predicting stability, making them less useful than DFT for searching. In particular, in chemical spaces where stable compounds are rare, only the model using crystal structure could efficiently tell which materials are stable. The paper concludes that high accuracy in formation energy does not imply high accuracy in stability Sourced [Reference 5].

The reason is the difference of scale. According to the same paper, for Materials Project compounds ΔHf spreads widely, with a mean of -1.42 eV/atom and a mean absolute deviation of 0.95 eV/atom, whereas ΔHd gathers in a much narrower range, with a mean of 0.06 eV/atom and a mean absolute deviation of 0.12 eV/atom [Reference 5]. A prediction error of several tens of meV/atom in ΔHf looks small compared with the whole spread of ΔHf. But compared with the spread of ΔHd (0.12 eV/atom), an error of the same size is enough to cross the boundary between stable and unstable (commentary).

(2) Checking the relation between error and decisions with synthetic data

We reproduced how small errors still give wrong decisions with synthetic data that you can check by hand.

• Data (made up): the distribution of the distance to the hull was made as a Laplace distribution matched to the report of Bartel and co-workers (mean 0.06, mean absolute deviation 0.12 eV/atom). If values of 0 eV/atom or below are "stable", about 30% are stable.
• Prediction: a Gaussian error was added to the true value. The size of the error is a mean absolute error (MAE) of 0.02 to 0.10 eV/atom.
• Decision: a prediction of 0 eV/atom or below is judged "stable", and precision and recall are calculated.

Result: with an MAE of 0.02 eV/atom, precision was 0.91 and recall 0.93. With an MAE of 0.06 eV/atom (half the spread of ΔHd), precision fell to 0.73 and recall to 0.82, and with an MAE of 0.10 eV/atom precision fell to 0.61 Our calculation. In other words, even with an apparently small error of "MAE 0.06 eV/atom", more than a quarter of the candidates called "stable" are misses.

Our calculation. Python (NumPy), 400,000 points, random seed 0. A simple assumption in which the error is added directly to the distance to the hull; these are not the results of any real model. In reality the distribution of errors is not Gaussian and is biased by chemical system (commentary).

(3) Knowing the "scale" of errors: experiments disagree with one another too

A sense of scale is also useful. Kirklin and co-workers compared DFT values in the OQMD database with 1,670 experimental formation energies. The mean absolute error between DFT and experiment was 0.096 eV/atom. They also examined, for compounds with several experimental sources, the mean absolute error between experiments themselves, and found a surprisingly large disagreement of 0.082 eV/atom. The authors say that a significant fraction of the difference between DFT and experiment may be due to experimental uncertainty Sourced [Reference 6].

So the experimental values taken as the "correct answer" already contain a disagreement of about 0.08 eV/atom. There is a region in which making the error of machine learning smaller than that cannot be called "closer to measurement" (commentary). The boundary of the stability decision (the spread of the hull distance of 0.12 eV/atom) also deals with differences of the same order.

5. How test data are built: splits, leakage, distribution shift

The basics of splitting (random split, split by group, data leakage) were covered with small examples in our explainer on machine learning and neural networks and in our explainer on applications to alloys and structural materials. Here we organize the circumstances specific to materials and the levels of how test data are built.

(1) Many similar records: why a "random split" becomes optimistic

Meredig and co-workers pointed out, in the context of materials discovery, that conventional machine-learning metrics overestimate performance for discovery. They proposed cross-validation that holds out whole clusters of similar items (LOCO CV) and a simple nearest-neighbour method as a baseline, and showed that performance in discovery depends strongly on the problem, the way data are sampled and extrapolation Sourced [Reference 7].

(2) Performance drops when the release changes: distribution shift

Li and co-workers showed that models trained on the 2018 release of the Materials Project can show severely degraded performance on the compounds newly added in the 2021 release. The point is that an excellent benchmark score does not imply good generalization. As countermeasures they present UMAP (a way to see the relation between training and test data in feature space) and the disagreement among several models (query by committee) to find data outside the distribution, and report that adding only 1% of the test data to training greatly improved prediction accuracy Sourced [Reference 8].

How to read it: a database changes as new calculations are added. It is necessary to record "which release of the data was used for training and which for evaluation" (commentary).

(3) Data leakage: learning from outside materials science

Kapoor and Narayanan surveyed 17 fields that use machine learning in science, found that data leakage affected at least 294 papers, and classified leakage into eight types. They propose "model info sheets" for reporting, and show a case in which, after correcting the leakage, the claim that complex machine learning beats a simple classical model no longer held Sourced [Reference 14]. We did not check whether the surveyed fields include materials science, but leakage of the same types can occur in materials MI: duplicated compounds, the same chemical system, the order of preprocessing and so on (commentary).

(4) Prospective tests: coming closer to real use

Matbench Discovery places prospective testing, evaluation on "data newly generated by the real search workflow", at the centre of its design [Reference 2]. But its "answer" is the DFT distance to the convex hull, not a value confirmed stable by experiment. A prospective test is an evaluation in the world of calculation that is closer to real use, and it still has to be confirmed by experiment in the end (commentary).

6. Reproducibility: what counts as "the same" in calculation and experiment

When we say "reproducible", the meaning differs depending on what is kept the same and what is changed. In materials MI the following three are easily mixed up, so we think of them separately (our own organization).

• The same procedure gives the same result: do the same data, code, settings and random seed give the same number? In machine-learning calculations, if the software version, data version and random seed are not recorded, you cannot reproduce it even yourself.
• Another tool gives the same result: do another calculation code or pseudopotential, another laboratory or instrument, give the same number?
• Other data give the same trend: do other samples or another set of compounds give the same accuracy? (This is the "generalization" of the previous chapter.)

(1) Reproducibility of experiments, in the words of measurement

The international vocabulary of metrology (VIM, JCGM 200:2012) defines measurement reproducibility as "measurement precision under reproducibility conditions". Reproducibility conditions are conditions that include different locations, operators and measuring systems, with replicate measurements on the same or similar objects. Repeatability is the precision when the measurement is repeated under the same conditions (repeatability conditions) Sourced [Reference 19]. ISO 5725-1:2023 is a standard that sets out the conditions and the statistical model needed to evaluate the "trueness" and "precision" of a measurement method, and ISO 5725-2:2025 gives the basic method for determining the repeatability and reproducibility of a standard measurement method [References 17, 18] (the full texts are paid, so we stay with the titles and scope).

In materials experiments, "the same person measures the same sample again on the same instrument" is repeatability, and "measuring in another laboratory, by another person, on another instrument" is reproducibility. The experimental benchmarks of JARVIS-Leaderboard are built on this idea of interlaboratory comparison [Reference 4].

(2) Reproducibility of calculation: even the same DFT changes with the code

Lejaeghere and co-workers computed the equation of state of 71 elemental crystals with 15 widely used DFT codes and 40 potentials, and compared them. The calculated values showed differences, but most recent codes and methods converged toward almost the same value, with differences comparable to experimental error Sourced [Reference 15]. This shows that "calculated by DFT" alone can still leave small differences depending on the code and potential, and that, when used properly, agreement between codes has reached the level of experimental error.

The electronic-structure area of JARVIS-Leaderboard provides a place to compare methods, software, pseudopotentials, materials and properties against experiment [Reference 4]. When reading a paper, check whether the code name and version, the functional, the pseudopotential, the cutoff and the k-points are given (commentary). That the choice of functional changes the band-gap error was seen in our explainer on applications to semiconductor and electronic materials.

(3) Reproducibility of machine learning: how it is reported

The "model info sheets" proposed by Kapoor and Narayanan are a reporting form that asks for an explanation of splits, preprocessing and duplicates for each of the eight types of data leakage [Reference 14]. In reports of materials MI, the following records, in addition, make it easier for others to check (commentary): the source and release of the data (for example the release of the Materials Project), the unit of the split, the order of preprocessing, software versions, random seeds and the number of repeats, and the definitions of the metrics.

7. Reading "AI discovered": claims, criticism and corrections

The phrase "AI discovered X thousand new materials" draws the most attention in materials MI and is also the most easily misunderstood. Here we line up, neutrally, two major published claims (GNoME and A-Lab) and the scholarly criticism and corrections that followed. We do not judge which is right and we did not replicate them. Based on the published documents, we organize "at which level the claim is made".

(1) GNoME: 2.2 million structures and 736 experimental realizations

Merchant and co-workers reported that graph neural networks trained at scale raised the efficiency of the search for stable crystals by an order of magnitude. Building on 48,000 stable crystals found in earlier studies, they state that they discovered 2.2 million structures below the current convex hull. They also say that, "of the stable structures, 736 have already been independently experimentally realized" Sourced [Reference 9]. "Stable" here is a classification based on the DFT convex hull, and the 2.2 million is a count of computed predictions. The 736 are structures that had been synthesized separately by other researchers and can be read as examples in which a prediction corresponded to a real substance (commentary).

In response, Cheetham and Seshadri examined the report from a chemistry viewpoint and wrote that "there is scant evidence for compounds that fulfil the trifecta of novelty, credibility and utility". At the same time they said the methods hold promise and that expertise in materials synthesis and crystallography needs to be incorporated Sourced [Reference 12]. We could not find a formal rebuttal of this point by the authors of GNoME.

(2) A-Lab: the success rate of an autonomous laboratory, and the meaning of "novel"

Szymanski and co-workers reported that the autonomous laboratory A-Lab for solid-state synthesis, over 17 days of continuous operation, planned experiments using computation, literature data, machine learning and active learning, and carried them out with robots. The abstract at publication said it realized "41 novel compounds from a set of 58 targets". An author correction published on 19 January 2026 acknowledges that concerns were raised about the unambiguous identification of the compound structures using diffraction and about the claims of novelty, and that the claims of novelty were subject to misinterpretation. The authors explain that the intention was "new to the prediction platform", not necessarily "new to science", and updated the text. The updated abstract says it realized "36 compounds from a set of 57 targets" Sourced [References 10, 11]. We did not check the details of why the numbers changed (an annotated PDF accompanying the correction).

Leeman and co-workers, using the 2023 report on A-Lab as an example, examined problems that can arise in unsupervised materials discovery. They examined all 43 synthesis products and argue that there are four common shortfalls in the analysis, which lead to the conclusion that "no new materials have been discovered in that work". Their main points are that automated Rietveld analysis of powder X-ray diffraction data is not yet reliable, and that while predictions put the elements on distinct crystallographic positions, in reality elements often share positions, giving higher-symmetry structures that are known alloys or solid solutions (neglect of disorder). They state that two thirds of the claimed successful materials are likely to be known compositionally disordered versions of the predicted ordered compounds Sourced [Reference 13]. This is the analysis of that paper's authors, and we did not verify it independently.

All Sourced (statements in each paper and correction, checked 22 October 2026). The arrangement of the table is Our calculation (organization). We did not check the difference between counting "41" for A-Lab and "43" in Leeman et al.

8. A materials engineer's view: your own evaluation and a question list

We summarize what we have covered as six questions to ask when reading a paper or a talk. The same questions can be used when you evaluate MI results in your own company (our own organization, not an official guideline).

9. Limits, what we could not confirm / summary

(1) Depth of checking of the papers

We checked the text of the papers by Dunn and co-workers, Riebesell and co-workers and Bartel and co-workers. We checked the papers by Merchant and co-workers, Szymanski and co-workers and its author correction, Cheetham and Seshadri, Leeman and co-workers, Kirklin and co-workers, Li and co-workers, Meredig and co-workers, Kapoor and Narayanan and Lejaeghere and co-workers within the abstract (or a published summary). For the JARVIS-Leaderboard paper we checked the abstract and the official page, and for the VIM the entries of the terms.

(2) The discussion on "AI discovered"

The claims of GNoME and A-Lab, the criticism of them and the author correction of A-Lab are the statements of published documents set side by side, and we did not replicate them. We could not confirm the annotated PDF accompanying the A-Lab author correction, a rebuttal by the authors of GNoME to Cheetham and Seshadri, or a rebuttal by the authors of A-Lab to Leeman and co-workers. We also did not check the difference between "41" for A-Lab and "43" in Leeman and co-workers.

(3) Changes in leaderboards and metrics

The leaderboard of Matbench Discovery and the weights of its combined score CPS are as of the checking date (22 October 2026). We give no ranking of specific models. We confirmed from the Crossref record that an Author Correction to the Matbench Discovery paper (2025) has been published, but we did not check its content.

(4) The reality of data leakage in materials

We could not confirm whether the fields surveyed by Kapoor and Narayanan include materials science. We could not find primary sources that systematically examined the frequency of leakage in materials MI.

(5) Standards and guidance

We could not find an international standard (ISO/IEC or similar) that defines the evaluation or reproducibility of materials MI. The full texts of ISO 5725 and the ISO/IEC standards are paid, so we confirmed only titles and scope.

(6) What we did not cover

Evaluation of language models (our explainer on benchmarks and how to read evaluations), evaluation of catalysts (Open Catalyst and others), individual evaluations of machine-learning interatomic potentials (our explainer on machine-learning interatomic potentials) and quantification of uncertainty (our explainer on uncertainty and applicability domain) are not covered.

10. How to read the yardsticks and standards

Results of materials MI are read with several yardsticks: error metrics (MAE, R² and so on), decision metrics (precision, recall, F1, the discovery acceleration factor and so on), the split, reproducibility and confirmation by experiment. Error metrics show "how far it misses on average", decision metrics "how well hit-or-miss judgements are right", the split "whether it also holds for new materials", reproducibility "whether the result is the same with another tool or at another place", and experimental confirmation "whether the material could really be made and measured". Because this article deals with evaluation itself, this chapter overlaps a great deal with earlier chapters. Here we put the benchmarks, standards and principles we could confirm into one table. We could not find an international standard specific to materials MI. The checking date is 22 October 2026.

The arrangement of the table is Our calculation (organization). Numbers, titles and publication dates of standards were checked on the official ISO pages (checked 22 October 2026). The full texts are paid, so we stay with the scope statements. Benchmarks and leaderboards are community efforts, not standards. ASTM standards are not covered in this article, and we did not consult the ASTM site.

Cautions when reading (for materials engineers)

• A benchmark ranking and your own material's strength are different: a leaderboard gives numbers for specific data, splits and metrics. Combined scores with different weights change the ranking if you emphasize different things. Re-measure on your own data.
• Prediction and experimental verification are different questions: even with a high score on judging stability, whether the material can be synthesized, whether the structure is right, whether it is new and whether it is useful are separate (chapter 7).
• Data leakage and the pitfall of splitting: if similar materials fall on both sides, numbers become optimistic. Performance can also drop when the release changes.
• Read standards and principles as recommended or required: FAIR is a set of principles (recommended) and the NIST AI RMF is a voluntary framework. An ISO standard is generally understood to become binding only when cited in laws or contracts (commentary). For "complies with the standard", check which requirement of which standard.
• Plans and achievements differ: leaderboards keep being updated and standards keep being revised (a new edition of ISO 5725-2 was published in December 2025). When you quote, add the edition and the checking date.

11. Glossary

12. References (primary sources)

1. Dunn A., Wang Q., Ganose A., Dopp D., Jain A. "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020), DOI 10.1038/s41524-020-00406-3 (open access, full text checked) - doi.org/10.1038/s41524-020-00406-3
2. Riebesell J., Goodall R. E. A., Benner P., Chiang Y. et al. "A framework to evaluate machine learning crystal stability predictions", Nature Machine Intelligence 7, 836-847 (2025), DOI 10.1038/s42256-025-01055-1 (full text checked; an Author Correction exists in the same journal, DOI 10.1038/s42256-025-01117-4) - doi.org/10.1038/s42256-025-01055-1
3. Matbench Discovery official website (leaderboard and metrics, checked 22 October 2026) - matbench-discovery.materialsproject.org
4. Choudhary K., Wines D., Li K., Garrity K. F. et al. (37 authors) "JARVIS-Leaderboard: a large scale benchmark of materials design methods", npj Computational Materials 10, 93 (2024), DOI 10.1038/s41524-024-01259-w (abstract checked; official page pages.nist.gov/jarvis_leaderboard) - doi.org/10.1038/s41524-024-01259-w
5. Bartel C. J., Trewartha A., Wang Q., Dunn A., Jain A., Ceder G. "A critical examination of compound stability predictions from machine-learned formation energies", npj Computational Materials 6, 97 (2020), DOI 10.1038/s41524-020-00362-y (open access, full text checked) - doi.org/10.1038/s41524-020-00362-y
6. Kirklin S., Saal J. E., Meredig B., Thompson A., Doak J. W., Aykol M., Rühl S., Wolverton C. "The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies", npj Computational Materials 1, 15010 (2015), DOI 10.1038/npjcompumats.2015.10 (abstract checked) - doi.org/10.1038/npjcompumats.2015.10
7. Meredig B., Antono E., Church C. et al. "Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery", Molecular Systems Design & Engineering 3, 819-825 (2018), DOI 10.1039/C8ME00012C (abstract checked) - doi.org/10.1039/C8ME00012C
8. Li K., DeCost B., Choudhary K., Greenwood M., Hattrick-Simpers J. "A critical examination of robustness and generalizability of machine learning prediction of materials properties", npj Computational Materials 9, 55 (2023), DOI 10.1038/s41524-023-01012-9 (abstract checked) - doi.org/10.1038/s41524-023-01012-9
9. Merchant A., Batzner S., Schoenholz S. S., Aykol M., Cheon G., Cubuk E. D. "Scaling deep learning for materials discovery", Nature 624, 80-85 (2023), DOI 10.1038/s41586-023-06735-9 (abstract checked) - doi.org/10.1038/s41586-023-06735-9
10. Szymanski N. J., Rendy B., Fei Y. et al. (16 authors) "An autonomous laboratory for the accelerated synthesis of inorganic materials", Nature 624, 86-91 (2023), DOI 10.1038/s41586-023-06734-w (abstract at publication and updated abstract after the correction checked) - doi.org/10.1038/s41586-023-06734-w
11. Szymanski N. J. et al. "Author Correction: An autonomous laboratory for the accelerated synthesis of inorganic materials", Nature 650, E1 (19 January 2026), DOI 10.1038/s41586-025-09992-y - doi.org/10.1038/s41586-025-09992-y
12. Cheetham A. K., Seshadri R. "Artificial Intelligence Driving Materials Discovery? Perspective on the Article: Scaling Deep Learning for Materials Discovery", Chemistry of Materials 36, 3490-3495 (2024), DOI 10.1021/acs.chemmater.4c00643 (abstract checked) - doi.org/10.1021/acs.chemmater.4c00643
13. Leeman J., Liu Y., Stiles J., Lee S. B., Bhatt P., Schoop L. M., Palgrave R. G. "Challenges in High-Throughput Inorganic Materials Prediction and Autonomous Synthesis", PRX Energy 3, 011002 (2024), DOI 10.1103/PRXEnergy.3.011002 (abstract checked) - doi.org/10.1103/prxenergy.3.011002
14. Kapoor S., Narayanan A. "Leakage and the reproducibility crisis in machine-learning-based science", Patterns 4, 100804 (2023), DOI 10.1016/j.patter.2023.100804 (open access, abstract checked) - doi.org/10.1016/j.patter.2023.100804
15. Lejaeghere K. et al. (69 authors) "Reproducibility in density functional theory calculations of solids", Science 351, aad3000 (2016), DOI 10.1126/science.aad3000 (summary text checked) - doi.org/10.1126/science.aad3000
16. Wilkinson M. D. et al. "The FAIR Guiding Principles for scientific data management and stewardship", Scientific Data 3, 160018 (2016), DOI 10.1038/sdata.2016.18 - doi.org/10.1038/sdata.2016.18
17. ISO "ISO 5725-1:2023 Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions" (ISO page) - iso.org/standard/69418.html
18. ISO "ISO 5725-2:2025 Accuracy (trueness and precision) of measurement methods and results - Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method" (ISO page) - iso.org/standard/90054.html
19. JCGM "JCGM 200:2012 International vocabulary of metrology - Basic and general concepts and associated terms (VIM)" (BIPM page; entries 2.21, 2.24 and 2.25 checked) - doi.org/10.59161/JCGM200-2012
20. ISO/IEC "ISO/IEC 5259-1:2024 Artificial intelligence - Data quality for analytics and machine learning (ML) - Part 1: Overview, terminology, and examples" (ISO page) - iso.org/standard/81088.html
21. ISO/IEC "ISO/IEC 22989:2022 Information technology - Artificial intelligence - Artificial intelligence concepts and terminology" (ISO page) - iso.org/standard/74296.html
22. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1, January 2023, DOI 10.6028/NIST.AI.100-1 (statement that it is intended for voluntary use checked) - doi.org/10.6028/NIST.AI.100-1

13. Claim-to-source audit

Last updated: 22 October 2026 / Sources are limited to primary ones (peer-reviewed papers, arXiv preprints, official websites, and official ISO, BIPM and NIST materials). The full texts of ISO standards are paid and were not checked. The synthetic-data example is a calculation on data made up for this article, not the performance of real materials. The discussion on "AI discovered" sets out the statements of published documents neutrally, and we did not replicate it. The content of some corrections and rebuttals, the reality of data leakage in materials, and a standard specific to evaluating materials MI could not be confirmed and are therefore not stated. The figures are explanatory concept diagrams, drawings that include our own calculation, or plots of published numbers; Figs. 1 and 7 are ImageGen-generated conceptual illustrations.