1. The short version, what to know first, and where this fits in the series

• Explainable AI (XAI) is the technology of showing the "reasons" for a prediction in a form people can understand. The best known, SHAP, splits one prediction into "the contribution of each feature" Sourced[Ref. 1]. But this explains "how the model uses features", not "whether the feature is a cause in the material"
• Look-alike features split the importance. In our synthetic data (y is determined by x1 alone), adding an x2 that closely tracks x1 (correlation 0.99) gave x2 an average of 25% of the importance. Across 12 models with the same accuracy, the share of x1 varied from 60 to 81% Our calculation. "High importance" does not mean "cause"
• An explanation is a hypothesis. It becomes knowledge only after you check, in order, that it survives rebuilding the model, that it agrees with materials expertise, and that it holds in an experiment that fixes the other conditions Sourced[Ref. 2]

2. What explainability is: two roads

The demand to know "why" an AI predicted something is strong well beyond materials. In researchers' terms, interpretability and explainability are similar but slightly different words. Oviedo et al. review interpretable and explainable machine learning in materials science and chemistry and state that there are trade-offs among the understandability, completeness and scientific validity of an explanation Sourced[Ref. 2]. Roscher et al. list transparency, interpretability and explainability as the core elements of explainable machine learning for the natural sciences, and say that domain knowledge is a prerequisite for explanation Sourced[Ref. 3].

• A. Models that are transparent from the start: linear regression, decision trees, symbolic regression that searches for a short formula, and so on. The formula or the branches are the explanation. Rudin argues that for high-stakes decisions one should use models that are interpretable from the start rather than explaining black boxes afterwards Sourced[Ref. 4].
• B. Explaining a black box afterwards: for accurate models such as random forests, neural networks and GNNs, apply methods such as permutation importance or SHAP.

3. Three ways to get feature importance

Here are three common methods in brief. All of them examine "which features matter, and how much" from the model.

• Permutation importance: shuffle the column of one feature between rows of the data and see how much the predictions worsen. The more they worsen, the more the model is taken to rely on that feature. The importance measure inside random forests is known from Breiman's 2001 paper Sourced[Ref. 6]. Weakness: Strobl et al. point out that random-forest importance is biased toward correlated features, and give two reasons (correlated features are preferred when trees are built, and the unconditional shuffling scheme favours correlated features) Sourced[Ref. 7].
• Partial dependence: move only one feature, keep the others as in the data, and plot how the average prediction changes. Friedman's 2001 gradient-boosting paper presented tools for interpreting tree models Sourced[Ref. 8].
• SHAP: splits one prediction into "the contribution of each feature" from a baseline. Lundberg and Lee gave a framework that unifies six existing methods (additive feature importance) and showed that within this class there is a unique solution with desirable properties Sourced[Ref. 1]. For tree models (random forests and so on) optimal explanations can be computed in polynomial time, and many local explanations can be combined to examine the global structure Sourced[Ref. 9]. LIME is another method, which explains a prediction by fitting a simple model around it Sourced[Ref. 10].

A small numerical example of SHAP

Let us check the idea of SHAP with a made-up "strength" formula. There are three factors, A (additive), B (hot firing) and C (coarse grains), and strength = 40 + 20×A + 15×B − 10×C + 12×A×B. The combination of A and B (an interaction) raises the strength by 12. We explain the sample with A=1, B=1, C=0 (prediction 87).

1. Baseline: the average prediction when the features take all eight combinations (55.5).
2. Splitting the contributions: how much each feature pushed the value from the baseline up to the prediction is found by averaging over all the "orders of entry" (the Shapley value). The result is +14.5 for A, +12.0 for B and +5.0 for C.
3. The interaction is shared: the 12 from the interaction of A and B is split half and half (6 each).
4. The sum matches: 14.5 + 12.0 + 5.0 = 31.5 equals the gap between prediction and baseline (87 − 55.5).

The contribution values are Our calculation (an exact calculation of a made-up formula, using eight independent combinations of the features as the baseline; not a formula for any real material).

4. Look-alike features split the importance

Material descriptors (the "business cards") include many that are similar to each other: for example ionic radius and lattice parameter, or electronegativity and an index built from it. What happens to an explanation when look-alike features are present? We checked with synthetic data.

• The data: y is determined by x1 alone (y = 3×x1 + noise). x2 is a copy of x1 made to resemble it (correlation ρ), and x3 is irrelevant. 300 training and 300 test samples. We trained a random forest and measured permutation importance on the test data, over 12 runs with different random seeds Our calculation.
• ρ = 0 (not similar): the share of x1 is 100%. It reads correctly.
• ρ = 0.9: x1 gets 93% and x2 7% (the share of x1 ranges from 89 to 96%) Our calculation.
• ρ = 0.99: x1 gets 75% and x2 25% (the share of x1 ranges from 60 to 81%) Our calculation. A quarter of the importance went to x2, which has nothing to do with y.
• The accuracy is the same: the 12 models at ρ = 0.99 have almost the same R², 0.955 to 0.968. Rebuilding without x2 gives a mean R² of 0.962, and the share of x1 returns to 100% Our calculation.

This agrees with what the literature has pointed out. Strobl et al. showed that random-forest importance is biased toward correlated features Sourced[Ref. 7]. Aas et al. state that the usual SHAP computation (Kernel SHAP), which assumes the features are independent, uses combinations of data that do not occur in reality when features are correlated, so that even for a simple linear model the explanation can be badly misleading, and propose an extension that accounts for dependence Sourced[Ref. 11]. Fisher et al. note that there are many models that fit the same data about equally well, and a feature that matters in one model may not matter in another Sourced[Ref. 12]. Molnar et al. list general pitfalls of model interpretation, including feature dependencies, interactions, uncertainty, high dimensions and unjustified causal interpretations Sourced[Ref. 13].

5. Correlation and causation: an explanation does not tell you the cause

The other big pitfall is mistaking correlation for causation. Take a made-up example. In one lab, there is a habit of adding more of an additive to samples fired at high temperature. It is the temperature that sets the strength; the additive has no effect. How does the data look?

• Overall, samples with more additive are stronger, with a correlation coefficient of 0.93. A model learns "the additive is important", and explanation methods say so too Our calculation.
• But when the data are split by firing temperature, the relation between additive and strength is almost flat (slopes of 3.2 at low temperature and 1.7 at high temperature, against 62 overall) Our calculation.
• As here, a correlation arises when another factor (here the firing temperature) affects both quantities. Such a factor is called a confounder.

Pearl explains that causal inference needs assumptions as well as data, and that the question "what happens if we change only one thing while leaving the others alone (an intervention)" is different from an observed correlation Sourced[Ref. 14]. Janzing et al. argue that measuring the relevance of features is a causal problem, which requires distinguishing conditional probabilities from interventions Sourced[Ref. 15]. Kumar et al. point to mathematical problems with Shapley-value importance and say that fixing them needs more complex machinery, such as causal reasoning Sourced[Ref. 16].

In their work on material descriptors, Ghiringhelli et al. state that when the scientific connection between a descriptor and the underlying mechanism is unclear, the causality of the learned relation is uncertain. They then show, with the classic example of the energy difference between zinc blende or wurtzite and rocksalt structures, how a meaningful descriptor can be found systematically Sourced[Ref. 17].

6. Checking against a materials engineer's expertise

We have summarised how to turn an explanation into "knowledge" in five questions.

Example: predicting phases of high-entropy alloys

Whether a solid solution (a single crystal phase in which several elements are mixed) forms has long been organised through materials knowledge. Zhang et al. (2008) summarised that the atomic size difference (δ) and the enthalpy of mixing determine the conditions for forming a solid solution, and stated that to form one an alloy should have a high entropy of mixing, a small δ, and an enthalpy of mixing that is neither strongly negative nor strongly positive Sourced[Ref. 18]. Guo et al. (2011) reported that the enthalpy of mixing is the key factor deciding between solid solutions and compounds, and that the stability of fcc and bcc solid solutions is well delineated by the valence electron concentration (VEC) Sourced[Ref. 19].

Machine-learning studies use the same kinds of features. Peivaste et al. (2023) compared machine-learning models on 5,692 experimental records covering 50 elements and 11 phase categories (XGBoost and random forests reached 86% accuracy over all phases) and state that feature importance varies across models and phases Sourced[Ref. 20]. This points the same way as our synthetic result. The viewpoints for checking an explanation against knowledge are collected in the table below.

The arrangement of the table is Our calculation (organisation). Alloy knowledge is commentary based on Zhang et al. [Ref. 18], Guo et al. [Ref. 19] and Peivaste et al. [Ref. 20]. Specific numerical thresholds for δ, VEC and so on are not covered here.

7. Choosing models that are interpretable from the start

Instead of "explaining afterwards", you can choose "a model that is interpretable from the start". SISSO (Ouyang et al., 2018), which builds on the descriptor work of Ghiringhelli et al., is a compressed-sensing method that finds the best low-dimensional descriptor (a short formula) among an immense number of candidate formulas Sourced[Ref. 21].

The arrangement of the table is Our calculation (organisation). Commentary based on Rudin [Ref. 4], Ouyang et al. [Ref. 21] and Molnar et al. [Ref. 13].

8. A materials engineer's view

9. Limits, what we could not confirm, and a summary

(1) What we could not confirm

• For Rudin's paper we keep to what the title says and did not confirm the detailed arguments in the text.
• We did not confirm the specific description of how importance is computed in Breiman (2001); we limit ourselves to saying that random-forest importance is known from that paper.
• For the studies by Zhang et al., Guo et al. and Peivaste et al. we stay within the abstracts and do not give specific thresholds or ranking figures.
• Our synthetic-data experiment is limited to three features, one target, a random forest and permutation importance. We did not compute SHAP values themselves on these data.
• We could not confirm a standard test for the fidelity of explanation methods for materials. The text of ISO/IEC TS 6254 is paid, and we did not check its requirements.

10. How to read the yardsticks and standards

The first yardstick for an explanation is fidelity (does the explanation reflect what the model actually does). Next come stability (is it the same when data or seeds change), consistency with knowledge (does it agree with what is known) and confirmation by intervention (was it tested by experiment). Even for a highly accurate model, the quality of the explanation must be checked separately. Checked on 22 October 2026.

The arrangement of the table is Our calculation (organisation). Numbers, titles and status were checked on the ISO and NIST pages (22 October 2026). The texts of the standards are paid, so we limit ourselves to titles and scope and do not state requirements.

Reading notes (for materials engineers)

• The quality of an explanation is separate from the accuracy of the model: models with the same accuracy give different explanations. Change seeds and models to check that the explanation is stable [Refs. 12, 20]
• NIST and ISO documents are recommendations or frameworks: the AI RMF is a "voluntary" framework and ISO/IEC TS 6254 is a technical specification. They generally gain binding force only when cited in laws or contracts (commentary) [Refs. 22, 23]
• Agreement with knowledge does not prove a cause: until it is confirmed by an intervention experiment it remains a hypothesis [Ref. 14]

11. Glossary

12. References

1. Lundberg S. M., Lee S.-I. "A unified approach to interpreting model predictions", Advances in Neural Information Processing Systems 30 (2017), arXiv:1705.07874 — arxiv.org/abs/1705.07874
2. Oviedo F. et al. "Interpretable and explainable machine learning for materials science and chemistry", Acc. Mater. Res. 3, 597–607 (2022), DOI 10.1021/accountsmr.1c00244 — doi.org/10.1021/accountsmr.1c00244
3. Roscher R. et al. "Explainable machine learning for scientific insights and discoveries", IEEE Access 8, 42200–42216 (2020), DOI 10.1109/ACCESS.2020.2976199 — doi.org/10.1109/ACCESS.2020.2976199
4. Rudin C. "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead", Nat. Mach. Intell. 1, 206–215 (2019), DOI 10.1038/s42256-019-0048-x — doi.org/10.1038/s42256-019-0048-x
5. Phillips P. J. et al. "Four principles of explainable artificial intelligence", NIST Interagency Report 8312 (September 2021), DOI 10.6028/NIST.IR.8312 — doi.org/10.6028/NIST.IR.8312
6. Breiman L. "Random forests", Mach. Learn. 45, 5–32 (2001), DOI 10.1023/A:1010933404324 — doi.org/10.1023/A:1010933404324
7. Strobl C. et al. "Conditional variable importance for random forests", BMC Bioinformatics 9, 307 (2008), DOI 10.1186/1471-2105-9-307 — doi.org/10.1186/1471-2105-9-307
8. Friedman J. H. "Greedy function approximation: a gradient boosting machine", Ann. Stat. 29, 1189–1232 (2001), DOI 10.1214/aos/1013203451 — doi.org/10.1214/aos/1013203451
9. Lundberg S. M. et al. "From local explanations to global understanding with explainable AI for trees", Nat. Mach. Intell. 2, 56–67 (2020), DOI 10.1038/s42256-019-0138-9 — doi.org/10.1038/s42256-019-0138-9
10. Ribeiro M. T., Singh S., Guestrin C. ""Why should I trust you?": explaining the predictions of any classifier", Proc. 22nd ACM SIGKDD, 1135–1144 (2016), DOI 10.1145/2939672.2939778 — doi.org/10.1145/2939672.2939778
11. Aas K., Jullum M., Løland A. "Explaining individual predictions when features are dependent: more accurate approximations to Shapley values", Artif. Intell. 298, 103502 (2021), DOI 10.1016/j.artint.2021.103502 — doi.org/10.1016/j.artint.2021.103502
12. Fisher A., Rudin C., Dominici F. ""All models are wrong, but many are useful": learning a variable's importance by studying an entire class of prediction models simultaneously", J. Mach. Learn. Res. 20(177), 1–81 (2019), arXiv:1801.01489 — arxiv.org/abs/1801.01489
13. Molnar C. et al. "General pitfalls of model-agnostic interpretation methods for machine learning models", Lecture Notes in Computer Science, 39–68 (2022), DOI 10.1007/978-3-031-04083-2_4 — doi.org/10.1007/978-3-031-04083-2_4
14. Pearl J. "Causal inference in statistics: an overview", Stat. Surv. 3, 96–146 (2009), DOI 10.1214/09-SS057 — doi.org/10.1214/09-SS057
15. Janzing D., Minorics L., Blöbaum P. "Feature relevance quantification in explainable AI: a causal problem", AISTATS 2020, arXiv:1910.13413 — arxiv.org/abs/1910.13413
16. Kumar I. E. et al. "Problems with Shapley-value-based explanations as feature importance measures", ICML 2020, arXiv:2002.11097 — arxiv.org/abs/2002.11097
17. Ghiringhelli L. M. et al. "Big data of materials science: critical role of the descriptor", Phys. Rev. Lett. 114, 105503 (2015), DOI 10.1103/PhysRevLett.114.105503 — doi.org/10.1103/PhysRevLett.114.105503
18. Zhang Y. et al. "Solid-solution phase formation rules for multi-component alloys", Adv. Eng. Mater. 10, 534–538 (2008), DOI 10.1002/adem.200700240 — doi.org/10.1002/adem.200700240
19. Guo S. et al. "Effect of valence electron concentration on stability of fcc or bcc phase in high entropy alloys", J. Appl. Phys. 109, 103505 (2011), DOI 10.1063/1.3587228 — doi.org/10.1063/1.3587228
20. Peivaste I., Jossou E., Tiamiyu A. A. "Data-driven analysis and prediction of stable phases for high-entropy alloy design", Sci. Rep. 13, 22556 (2023), DOI 10.1038/s41598-023-50044-0 — doi.org/10.1038/s41598-023-50044-0
21. Ouyang R. et al. "SISSO: a compressed-sensing method for identifying the best low-dimensional descriptor in an immensity of offered candidates", Phys. Rev. Mater. 2, 083802 (2018), DOI 10.1103/PhysRevMaterials.2.083802 — doi.org/10.1103/PhysRevMaterials.2.083802
22. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (26 January 2023), DOI 10.6028/NIST.AI.100-1" — doi.org/10.6028/NIST.AI.100-1
23. ISO/IEC "ISO/IEC TS 6254:2025 Information technology — Artificial intelligence — Objectives and approaches for explainability and interpretability of machine learning (ML) models and artificial intelligence (AI) systems", (ISO page) — iso.org/standard/82148.html
24. ISO/IEC "ISO/IEC 22989:2022 Information technology — Artificial intelligence — Artificial intelligence concepts and terminology", (ISO page) — iso.org/standard/74296.html

13. Claim-to-source audit

Last updated: 22 October 2026. Sources are limited to primary material (peer-reviewed papers, arXiv papers, and official NIST and ISO pages). The simulations and numerical examples (the random forest on synthetic data, the made-up SHAP example, the confounding data) are our own calculations, not results for real materials. We could not confirm the detailed arguments in Rudin's text, the specific way importance is computed in Breiman's paper, specific thresholds and rankings in the alloy studies, SHAP values on the synthetic data, or a materials-specific standard for measuring the fidelity of explanations, so we do not state them. All figures are concept diagrams or diagrams that include our own calculation; the ImageGen images are conceptual illustrations.