1. Three-line summary, prerequisites, and where this fits in the series

• A prediction should carry a “level of confidence.” Uncertainty comes in a kind that cannot be reduced, such as measurement scatter, and a kind that is lack of knowledge in ranges without data, which shrinks when you measure. What matters for deciding experiments is mainly the latter
• Uncertainty estimates can also be wrong. Even from the same 15 data points, some methods show an uncertainty band that widens where there are no data, and others stay narrow. In our example, the share of values that actually fell inside a “95% interval” outside the training range ranged from 24% (an ensemble of 3-nearest-neighbor models) to 92% (an ensemble of cubic polynomials)
• Check the applicability domain first. A prediction for a material far from the training data is an extrapolation. In our example, the error in the region far from the training points was more than 100 times that near them (0.015 → 1.92), and worse than a “return the mean” prediction

2. Two kinds of uncertainty: what can be reduced and what cannot

An analogy: if the search for materials is “looking for treasure on a very large map,” a prediction is a guess that “the treasure is probably here.” Without a note of “how sure,” you cannot decide whether to dig. That “how sure” is uncertainty. We think of uncertainty as two kinds.

View original SVG, values and sources

Original SVG reference: Conceptual diagram (vector drawing). Two kinds of uncertainty. Left: random error that stays however much data you add. Right: lack of knowledge where there are no data, so the model could give many answers; it shrinks as you measure. They are called aleatoric and epistemic uncertainty.

• Random error (aleatoric: noise inherent in the observation). The part that scatters every time even when you measure under the same conditions. It does not go away however much data you add. For example, the OQMD paper reports that the mean absolute error between experiments measuring the formation energy of the same compound is 0.082 eV/atom, “surprisingly large”Sourced [Ref. 3]. When comparing with experimental values, this gives a sense of why a model error far below this scatter should not be expected (commentary)
• Lack of knowledge (epistemic: uncertainty in the model). The part where the model could give many answers in ranges without data. It shrinks when you add data. Kendall and Gal distinguish the former as “noise inherent in the observations” and the latter as “uncertainty in the model, which can be explained away given enough data”Sourced [Ref. 1]. An introduction to these concepts from a machine-learning viewpoint is the article by Hüllermeier and Waegeman [Ref. 2]

For a materials engineer, what matters for deciding experiments is mainly the latter (lack of knowledge). “Where the prediction is uncertain” is where measuring adds the most information (Chapter 8).

The review by Ramprasad and colleagues says that MI predictions are typically interpolative but may also be extrapolative, and that then prediction uncertainties must be properly taken into accountSourced [Ref. 6].

3. How to estimate uncertainty: ensembles, Gaussian processes and conformal prediction

A typical machine-learning model returns one predicted value and no uncertainty. Here are three representative ways to obtain uncertainty. Each has a different idea and different strengths and weaknesses.

The “idea” and “caution” columns are our own commentary. We do not cover other methods, such as Bayesian approaches or methods using dropout.

In materials, there are reports of using uncertainty in experimental planning. Ling and colleagues, in optimizing composition and processing, give a model learned from data an uncertainty estimate and combine exploring high-uncertainty candidates with exploiting high-performing regions to propose the next experiment. Over four materials-science test cases, they report that the optimal candidate was found with on average three times fewer measurements than random guessingSourced [Ref. 12]. Our explainer on Bayesian optimization and active learning covers this in more detail.

4. Do uncertainty intervals hold inside and outside the range?

We check with a small experiment whether an uncertainty estimate is really useful (Our calculation). Let the true relation be y = sin(2x) + 0.3x, and train on 15 points (noise 0.1) measured in the range x = 0 to 5. From the same data we build prediction bands (prediction ± 2 standard deviations) with a Gaussian process and with an ensemble of 3-nearest-neighbor models (50 models built by bootstrap).

The left of Fig. 3 is the Gaussian process. In the range with data (blue background) the band is thin and the prediction overlaps the true relation (gray dotted line) well. Outside the range, the band widens greatly, honestly showing “we do not know here.” The right is the ensemble of 3-nearest-neighbor models. Even outside the range, the band stays narrow and the prediction stays flat at the edge value. The true relation (gray dotted line) moves a lot, yet the model says it is “sure.”

We look at the difference in numbers. As promised, we counted how many actual values fell inside the “95% interval,” inside and outside the range.

• Gaussian process: 95% inside the range (as promised), 81% outside
• Ensemble of 3-nearest-neighbor models: 86% inside, 24% outside. Outside, more than three times in four, the value was out of the interval
• Ensemble of cubic polynomials: 91% inside, 92% outside (it holds outside because the intervals are very wide, though whether that width is practical is another matter)

Our calculation. Synthetic data. The interval is prediction ± 1.96 × (the standard deviation of uncertainty and noise combined). Inside the range is x = 0 to 5; outside is x = −2 to 0 and 5 to 9. Mean of 200 redrawings of the data. The Gaussian-process kernel was fixed (RBF, length 1, variance 1, noise 0.1). These are not results on real materials and do not rank the methods.

5. Measuring the “quality” of uncertainty: calibration and sharpness

An uncertainty estimate has a quality. Tran and colleagues present a suite of figures and performance metrics from the machine-learning community for judging the quality of uncertainty estimates for materials property predictions, probing accuracy, calibration and sharpness quantitativelySourced [Ref. 13]. In plain words:

• Accuracy: how close the predicted values are to the true values
• Calibration: when it says “95%,” does it really hold 95% of the time? The table in Chapter 4 examined this calibration inside and outside the range
• Sharpness: how narrow the intervals are. An infinitely wide interval always holds but is useless. The best is narrow and still holds

Three related reports. Guo and colleagues report that modern neural networks, unlike those from a decade ago, are poorly calibrated, that depth, width, weight decay and batch normalization influence calibration, and that temperature scaling (a post-processing step that adjusts confidence with a single parameter) is surprisingly effective on most datasetsSourced [Ref. 14]. Hirschfeld and colleagues compared uncertainty-quantification methods on five regression datasets for molecular property prediction and report that none of the methods was unequivocally superior to all others, and none produced a particularly reliable ranking of errors across datasetsSourced [Ref. 15]. Kailkhura and colleagues point out that, with imbalanced data, standard methods for assessing the quality of models break down and lead to misleading conclusions, and that the model's own confidence score cannot be trustedSourced [Ref. 16].

6. Applicability domain: deciding where a prediction may be made

Separate from uncertainty, the idea of deciding whether a material lies inside or outside the range the model learned from is the applicability domain (AD). In the field of chemical structure-activity relationships (QSAR), defining the applicability domain is one of the principles for validating models. The paper by Sahigara and colleagues introduces the five OECD principles for validating (Q)SAR models as: (1) a defined endpoint, (2) an unambiguous algorithm, (3) a defined domain of applicability, (4) appropriate measures of goodness-of-fit, robustness and predictivity, and (5) a mechanistic interpretation, if possible. It explains that reliable predictions are generally limited to query chemicals that are structurally similar to the compounds used to build the modelSourced [Ref. 17].

There are several types of methods for deciding the domain. Sahigara and colleagues classify them by how they characterize the interpolation space in the descriptor space of the training data: range-based methods (such as a “box” of the minimum to maximum of each descriptor), geometric methods (such as the “convex hull” enclosing the training points), distance-based methods (such as the Mahalanobis distance) and probability-density-based methods [Ref. 17].

Fig. 5 overlays three ways of drawing it, from 60 training points of two correlated descriptors (correlation 0.9) (Our calculation).

• Box (orange dashed). It only encloses the minimum to maximum of each descriptor, so it also includes corners without training points. Query A is inside the box but far from the training points, so it is in fact an extrapolation. Of points scattered evenly in the box, only about 29% fell inside the convex hull
• Convex hull (green). The smallest convex region enclosing the training points. It is narrower than the box and closer to the real distribution. But as the number of descriptors grows, it becomes hard to compute, and it cannot represent holes between training points. Sahigara et al. also report that the convex hull and probability-density methods flagged the largest number of test compounds as outside the domain, allowing only limited use of the models [Ref. 17]
• Distance (blue dotted). An ellipse of points whose Mahalanobis distance (a distance that takes the correlation of descriptors into account) from the center of the training points is within the 95th percentile of the training points. In this example, about 35% of the box falls inside. Sahigara et al. report that distance-based approaches flagged a reasonable number of compounds as outside the domain, but that model statistics fell for some distance measures [Ref. 17]

Our calculation. Synthetic data. We counted, over 20,000 uniform random points in the box, the share that falls inside each method's domain. Not results on real materials. The conclusions of Sahigara et al. come from a comparison on models of the bioconcentration factor of chemicals and do not necessarily carry over to materials properties.

Here is another viewpoint. Sutton and colleagues point out that evaluating a model only by its average error over everything can make models whose average errors look the same actually differ greatly in performance across materials, and can hide that a model that looks insufficient overall works well in specific sub-domains. They propose a method based on subgroup discovery to find the domain of applicability from the data, and show for the prediction of formation energies of transparent conducting oxides that three state-of-the-art models with mutually indistinguishable average error have domains of applicability with distinctive featuresSourced [Ref. 18].

7. Distance from the training data, and error

We check with a small experiment whether the idea of an applicability domain is really useful (Our calculation). With two descriptors (range 0 to 2) we measure 60 points and train a Gaussian process. We evaluate on 2,000 points in the range 0 to 5, and for each point look at the distance to the nearest training point and the prediction error.

The results are in Fig. 6. When the distance to the nearest training point is under 0.1, the mean absolute error is 0.015. It rises with distance: 0.19 for 0.4 to 0.8, 0.55 for 0.8 to 1.6, and 1.92 for 1.6 or more. In the region of 1.6 or more, it is worse than the mean absolute error (1.27) of a prediction that just returns the mean of all data.

Our calculation. Synthetic data (true relation y = sin(1.5x₁) + 0.5cos(2x₂) + 0.3x₁x₂, noise 0.05). 60 training points (range 0 to 2), 2,000 test points (range 0 to 5). Gaussian process (RBF, fixed settings). Not results on real materials.

8. How to use it: deciding the next experiment

Uncertainty and applicability domain are computed together at every prediction and used in decisions. Fig. 7 gathers the flow (our own organization).

1. Predict. Predict the value of a new candidate with the model
2. Compute two things. The uncertainty of the prediction, and the distance to the training data (a guide to the applicability domain)
3. Inside the domain? Decide by distance, box, convex hull and so on. Overlaying several methods reduces what you miss
4. If inside, use with a level of confidence. Prefer candidates with small uncertainty and promising performance. But uncertainty is not zero
5. If outside, treat it as extrapolation. Either do not use it or measure it. Candidates outside lie in thinly covered regions, so measuring them adds much information
6. Check by experiment. A prediction is a candidate, and confirmation is by experiment
7. Add the results to the data. Update the model and the domain

Li and colleagues showed for several large materials datasets that up to 95% of the data can be removed from training with little effect on in-distribution performance, but that the degradation out of distribution is not prevented, and report that uncertainty-based active learning can construct much smaller but equally informative datasetsSourced [Ref. 21]. It suggests that where you measure matters more than how much data you have (commentary).

9. Limits, what we could not confirm, and a summary

(1) What we could not confirm

• Numerical experiments. The calculations in Chapters 4, 6 and 7 use synthetic data. They do not show how well uncertainty estimates hold, or how effective domain tests are, in real materials. Nor do they rank particular methods
• The original OECD principles. The OECD site could not be opened because of an automated-access restriction, so we used the introduction in the open-access paper by Sahigara et al. We did not confirm the adoption date or details
• Preconditions of the conformal-prediction guarantee. We introduced the abstract of the original paper (a guarantee without distributional assumptions) but did not confirm the details of the conditions under which it holds, nor the guarantee in extrapolation
• Other uncertainty methods. We do not cover Bayesian neural networks, methods using dropout, or methods that output a predictive distribution
• Which domain method is best. The comparison of Sahigara et al. is on models of chemicals. We did not confirm which method is best for predicting materials properties
• International standards on uncertainty. We could confirm those on measurement uncertainty (JCGM 100, ISO 5725-1), but could not confirm an international standard specific to the uncertainty of machine-learning predictions or to the applicability domain (Chapter 10)

10. How to read the yardsticks and standards

How good uncertainty and applicability domain are depends on which yardstick you use. The main yardsticks are the calibration of uncertainty (does “95%” really hold 95% of the time), sharpness (narrowness of intervals), differences in error inside and outside the range, extrapolation performance (forward cross-validation and the like) and measurement uncertainty (scatter on the experimental side). Each measures something different and leaves something out. For example, good calibration is useless if the intervals are too wide. The table lists what we could confirm (checked 22 October 2026). We could not confirm an international standard specific to the uncertainty and applicability domain of machine-learning predictions.

The arrangement of the table is Our calculation (organization). ISO/IEC and ISO numbers, titles and dates were checked on the official ISO pages (22 October 2026). The NIST AI RMF wording was checked in the document's text. Standard texts are paywalled, so we checked only titles and dates and wrote no requirements.

Cautions in reading (for materials engineers)

• Check calibration in a report that gives uncertainty. Even if it says “95% interval,” whether it really holds 95% of the time is another matter. See whether there is a check outside the range
• Trust applicability domains whose method is stated. Check that it says which descriptors, which distance (or box, or hull) and what threshold
• Do not mix measurement uncertainty with prediction uncertainty. Read with suspicion a claim that a model's error is smaller than the measurement scatter
• Most principles and guidelines are recommendations. The NIST AI RMF is meant for voluntary use. The OECD principles belong to the field of chemicals. ISO/IEC standards generally bind only when cited in law or contracts (commentary). When something says “compliant,” check which requirement of which document is meant

11. Glossary

12. References (primary sources)

1. Kendall A., Gal Y. "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?", arXiv:1703.04977 (NeurIPS 2017) — arxiv.org/abs/1703.04977
2. Hüllermeier E., Waegeman W. "Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods", Mach. Learn. 110, 457–506 (2021), DOI 10.1007/s10994-021-05946-3 — doi.org/10.1007/s10994-021-05946-3
3. Kirklin S. et al. "The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies", npj Comput. Mater. 1, 15010 (2015), DOI 10.1038/npjcompumats.2015.10 (open access) — doi.org/10.1038/npjcompumats.2015.10
4. JCGM "JCGM 100:2008 Evaluation of measurement data — Guide to the expression of uncertainty in measurement (GUM)" (PDF published by BIPM) — bipm.org/documents/…/JCGM_100_2008_E.pdf
5. ISO "ISO 5725-1:2023 Accuracy (trueness and precision) of measurement methods and results — Part 1: General principles and definitions" (ISO page) — iso.org/standard/69418.html
6. Ramprasad R. et al. "Machine learning in materials informatics: recent applications and prospects", npj Comput. Mater. 3, 54 (2017), DOI 10.1038/s41524-017-0056-5 — doi.org/10.1038/s41524-017-0056-5
7. Lakshminarayanan B., Pritzel A., Blundell C. "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles", arXiv:1612.01474 (NeurIPS 2017) — arxiv.org/abs/1612.01474
8. Wager S., Hastie T., Efron B. "Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife", J. Mach. Learn. Res. 15, 1625–1651 (2014) — jmlr.org/papers/v15/wager14a.html
9. scikit-learn User Guide "Gaussian Processes" — scikit-learn.org/stable/modules/gaussian_process.html
10. Rasmussen C. E., Williams C. K. I. "Gaussian Processes for Machine Learning", MIT Press (2006) (book web page) — gaussianprocess.org/gpml/
11. Angelopoulos A. N., Bates S. "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification", arXiv:2107.07511 — arxiv.org/abs/2107.07511
12. Ling J. et al. "High-Dimensional Materials and Process Optimization Using Data-Driven Experimental Design with Well-Calibrated Uncertainty Estimates", Integr. Mater. Manuf. Innov. 6, 207–217 (2017), DOI 10.1007/s40192-017-0098-z — doi.org/10.1007/s40192-017-0098-z
13. Tran K. et al. "Methods for comparing uncertainty quantifications for material property predictions", Mach. Learn.: Sci. Technol. 1, 025006 (2020), DOI 10.1088/2632-2153/ab7e1a — doi.org/10.1088/2632-2153/ab7e1a
14. Guo C. et al. "On Calibration of Modern Neural Networks", arXiv:1706.04599 (ICML 2017) — arxiv.org/abs/1706.04599
15. Hirschfeld L. et al. "Uncertainty Quantification Using Neural Networks for Molecular Property Prediction", J. Chem. Inf. Model. 60, 3770–3780 (2020), DOI 10.1021/acs.jcim.0c00502 — doi.org/10.1021/acs.jcim.0c00502
16. Kailkhura B. et al. "Reliable and explainable machine-learning methods for accelerated material discovery", npj Comput. Mater. 5, 108 (2019), DOI 10.1038/s41524-019-0248-2 (open access) — doi.org/10.1038/s41524-019-0248-2
17. Sahigara F. et al. "Comparison of Different Approaches to Define the Applicability Domain of QSAR Models", Molecules 17, 4791–4810 (2012), DOI 10.3390/molecules17054791 (open access) — pmc.ncbi.nlm.nih.gov/articles/PMC6268288/
18. Sutton C. et al. "Identifying domains of applicability of machine learning models for materials science", Nat. Commun. 11, 4428 (2020), DOI 10.1038/s41467-020-17112-9 (open access) — doi.org/10.1038/s41467-020-17112-9
19. Meredig B. et al. "Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery", Mol. Syst. Des. Eng. 3, 819–825 (2018), DOI 10.1039/C8ME00012C — doi.org/10.1039/C8ME00012C
20. Xiong Z. et al. "Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation", Comput. Mater. Sci. 171, 109203 (2020), DOI 10.1016/j.commatsci.2019.109203 — doi.org/10.1016/j.commatsci.2019.109203
21. Li K. et al. "Exploiting redundancy in large materials datasets for efficient machine learning with less data", Nat. Commun. 14, 7283 (2023), DOI 10.1038/s41467-023-42992-y (open access) — doi.org/10.1038/s41467-023-42992-y
22. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1, January 2023, DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1
23. ISO/IEC "ISO/IEC 23053:2022 Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)" (ISO page) — iso.org/standard/74438.html

13. Claim-to-source audit

Last updated: 22 October 2026 / Sources are limited to primary materials (peer-reviewed papers, arXiv papers, and official pages of scikit-learn, NIST, ISO and BIPM). The full text of ISO/IEC standards is paywalled and was not checked. The OECD principles are taken from an introduction in an open-access paper. The numerical experiments in Chapters 4, 6 and 7 are calculations on synthetic data we generated; they are not results on real materials and do not rank methods. We could not confirm the original OECD principles, details of the conformal-prediction guarantee, which domain method is best for materials, or an international standard specific to the uncertainty of machine-learning predictions, so these are not stated. All figures are explanatory concept diagrams or drawings that include our own calculation; the hero image and Fig. 8 are ImageGen-generated conceptual illustrations.