1. Three-line summary, prerequisites, and where this fits in the series

• With few data, it is easy to “look right.” R² on the training data is high, but new samples are missed. It breaks down especially when the number of descriptors approaches the number of data. The cross-validation number used for evaluation also swings widely when data are few
• The remedies are “careful evaluation” and “use of prior knowledge.” Narrow the descriptors, rein them in by regularization, pretrain on a large related dataset (transfer learning), and put physical and chemical knowledge into the descriptors. None of these is all-powerful
• The one point to remember: if you “select” descriptors or “augment” data before splitting the data, the evaluation number looks better only in appearance (a leak). In our synthetic data, even pure noise gave an R² of +0.45

2. How “small” are materials data?

An analogy: if machine learning is “an approximator that learns trends from past data” and the search for materials is “looking for treasure on a very large map,” small data is a map with only a few marks on it. From those few marks you have to guess the rest of the map. With so few marks, a chance shift in position changes the guess a great deal.

Zhang and Ling note that materials datasets are typically smaller, and sometimes more diverse, than in other fields, and studied how the amount of data affects prediction. The effect of data size is not on precision directly but is mediated by the model's degree of freedom (the number of things it can adjust). The appearance of a precision-versus-degree-of-freedom association signals underfitting (too simple even to capture the trend), characterized by a large bias of prediction, which restricts accurate prediction in unknown domainsSourced [Ref. 1]. The table gives examples of data sizes in materials machine learning that we could confirm in primary sources.

The counts in the table are those of each paper under its conditions. Size differs widely by material and property, and there is no single line for “small data.” Dunn et al. report that crystal graph methods appear to outperform traditional machine learning given about 10⁴ or more data points [Ref. 2].

Let us see what happens when data are few, with a small experiment (Our calculation). Of ten descriptors, only three really matter; the rest are noise. We vary the number of training data from 12 to 300 points and measure R² on new data for least squares (the most basic fit, which minimizes the sum of squared errors) and ridge regression (a way of reining in the descriptors; Chapter 6).

The results are in Fig. 2. With 12 points (close to the 10 descriptors), least squares has an R² of 0.98 on the training data, nearly perfect, but −6.0 on new data, a huge miss. With 15 points it is 0.27, with 20 points 0.65, with 30 points 0.74 and with 300 points 0.83, approaching the theoretical ceiling (about 0.84) as the data grow. The gap between the fit on training data and on new data is largest for small data. Ridge regression gets 0.48 even with 12 points and fails less badly.

Our calculation. Synthetic data (y = 2x₁ − x₂ + 0.5x₃ + noise with standard deviation 1; ten descriptors, standard normal). Each point is a mean over 200 redrawings of the data. Ridge strength was fixed at n/10 (minimum 1), not tuned. Not results on real materials.

3. Why too many descriptors break a model

One of the biggest pitfalls of small data is that the number of descriptors approaches the number of data. The more items you write on the card, the more likely some happen to move like the property by chance. The model uses them as clues and can explain the training data perfectly.

We check with a small experiment (Our calculation). There are 20 training points and only two descriptors really matter. We keep adding noise descriptors.

The results are in Fig. 3 (median R²). With only the two meaningful descriptors, R² on new data is 0.81, close to the ceiling (about 0.83). With 10 descriptors it is 0.65, with 15 it is 0.32, and with 19 (= the number of data minus 1) it is −4.7: a collapse. With 19 descriptors, least squares fits the training data perfectly (R² = 1.0) yet predicts nothing on new samples. Ridge regression (with its strength chosen by 5-fold cross-validation) limited the damage: 0.52 with 15 descriptors and 0.42 with 19.

Our calculation. Synthetic data (y = 2x₁ − x₂ + noise; the other descriptors are noise). 20 training points, 3,000 test points, median of 150 runs. With 40 and 100 descriptors, least squares (the minimum-norm solution) gave 0.23 and 0.12, and ridge 0.20 and 0.03, all far below the ceiling (the behavior in this region is related to a phenomenon called double descent, touched on in our explainer on the basics of machine learning and neural networks).

4. Cross-validation: with few data, even the evaluation number swings

Evaluation is done on data the model has not seen. The scikit-learn user guide explains that learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data [Ref. 11].

When data are few, setting aside a large test set is wasteful, so we use cross-validation. With 5 folds, for example, you split the data into five parts, train on four and evaluate on the remaining one, repeat five times, and average the evaluations. But the fewer the data, the more this average swings. Varoquaux points out that small sample sizes lead to large error bars in cross-validation, and that the standard error across folds strongly underestimates themSourced [Ref. 12].

We check with a small experiment (Our calculation). With the same mechanism, we redraw the data 300 times, compute the 5-fold cross-validation R² each time, and look at the spread of values for each data size.

The results are in Fig. 4. With 20 points, the cross-validation R² swung between 0.35 and 0.88 (5th to 95th percentile). The mean was 0.69, a little more modest than the true mean on new data (0.77). With 50 points it was 0.71 to 0.89, and with 200 points 0.79 to 0.87, a narrowing spread. With the same method and the same mechanism, the cross-validation number differs this much depending on which data happened to be drawn.

Our calculation. Synthetic data (3 of 5 descriptors informative, with noise). Least squares. Data redrawn 300 times. This spread comes from “how the data were drawn”; repeating cross-validation on the same data with different random splits (such as RepeatedKFold) does not shrink it. Not results on real materials.

Three habits so you do not lean too hard on the evaluation number

• Watch the spread. Do not decide which method is better from one cross-validation number. Where possible, repeat with different splits (scikit-learn's RepeatedKFold [Ref. 11]) and check the spread. As shown above, though, the uncertainty of how the data were drawn is not reduced by repeating
• Separate tuning from evaluation (nested cross-validation). If you choose hyperparameters (settings a person decides before learning, such as ridge strength) by cross-validation and then report the same number as your score, the score is too generous. Varma and Simon showed that the cross-validation error estimate for a classifier with optimized parameters was a substantially biased estimate of the true error on independent data, even for “null” data with no real difference, and that a nested cross-validation (tune in the inner loop, evaluate in the outer loop) is a remedySourced [Ref. 14]. Cawley and Talbot also report that the variance of the evaluation estimate leads to over-fitting in model selection, with effects comparable in size to differences between learning algorithmsSourced [Ref. 13]
• Split by group. If repeated measurements of one sample, or samples of the same lot, fall in both training and evaluation, the number is overstated. scikit-learn's GroupKFold is a way of splitting so that the same group is never in both training and evaluation [Ref. 11]

5. Two “leaks”: when to select features and when to augment data

With few data, the temptation to “somehow raise the score” is strong. In the process, information from the evaluation data can leak into training. Here are two leaks that small data are especially prone to.

(1) Selecting features before splitting the data

When there are many descriptors, you want to pick those that look useful. If you select descriptors using all the data and then run cross-validation, information from the evaluation leaks into the selection. Ambroise and McLachlan, working on a small number of tumor samples with expression data for a very large number of genes, showed that when gene selection is not external to the cross-validation, selection bias makes the error look too small, and recommended performing the cross-validation or the bootstrap external to the selection processSourced [Ref. 15]. The scikit-learn user guide likewise explains that preprocessing, such as standardization and feature selection, should be learned from the training set and only applied to held-out data [Ref. 11].

We check with a small experiment (Our calculation). With pure-noise data (40 samples, 500 descriptors, unrelated to the property), we choose the 10 descriptors most correlated with the property and measure R² by 5-fold cross-validation.

(2) Augmenting data before splitting it

To increase the data, there is a method of making copies of existing samples with added noise (data augmentation). If you augment first and then split at random into training and evaluation, copies of the same sample end up in both. The model simply memorizes the copies and scores high.

We check this too (Our calculation). We copy each of 30 samples 10 times (adding noise) and predict by the value of the closest point. The property is determined by one descriptor plus noise.

The results are in Fig. 5. Left: although the data are pure noise, selecting descriptors first on all the data gives a cross-validation R² of +0.45, which looks like a hit. With the correct procedure (selecting only inside each training fold) it is −0.48, correctly showing that nothing is predicted. Right: splitting at random after augmenting gives an R² of 1.00, near perfect, but splitting by original sample gives −0.31.

Our calculation. Synthetic data. Mean of 200 runs. The left case is pure noise, so a correct evaluation should be at or below 0. Not results on real materials. Kapoor and Narayanan report that data leakage affected at least 294 papers in 17 fields that use machine learning, and organize the types of leakageSourced [Ref. 16].

6. Regularization: reining in the descriptors

The basic remedy for small data is regularization. It adds a penalty so that the model's coefficients (the weights multiplied by the descriptors) do not become too large. The scikit-learn user guide explains ridge regression as addressing some problems of ordinary least squares by imposing a penalty on the size of the coefficients, where a larger complexity parameter α means more shrinkage and coefficients that are more robust to collinearity (several similar descriptors) [Ref. 10]Sourced. The original ridge-regression paper is Hoerl and Kennard (1970) [Ref. 8].

As a formula, it minimizes “the sum of squared errors” plus “α times the sum of squared coefficients.” In words: you want to make the error small, but forcing the fit with large coefficients costs you α per unit. If α is 0, it is ordinary least squares. As α grows, the model becomes simpler (smaller coefficients).

The lasso changes the form of the penalty so that some coefficients become exactly zero. scikit-learn explains that the lasso can set coefficients exactly to zero and tends to prefer solutions with fewer non-zero coefficients, effectively reducing the number of features the solution depends on [Ref. 10]Sourced. The original is Tibshirani (1996) [Ref. 9]. As a rule of thumb: ridge “keeps all descriptors and shrinks them”; the lasso “drops useless descriptors by setting them to 0” (commentary).

In Fig. 3, ridge regression improved least squares from 0.32 to 0.52 with 15 descriptors, and from −4.7 to 0.42 with 19 (Our calculation). But it is not always all-powerful. With still more descriptors, ridge fell to 0.03 (100 descriptors). The value of α is a hyperparameter that a person decides; it is chosen by cross-validation, but the fewer the data, the more that choice also swings (Chapter 4).

7. Transfer learning: borrowing the strength of related data

Even if your own measurements are few, a large related dataset (for example, public data computed by DFT) may be available. Using it for pretraining first, and then carrying the result over to the small-data task, is transfer learning. Here are some studies.

• XenonPy.MDL by Yamada et al. They released a library of more than 140,000 pretrained models for various properties of small molecules, polymers and inorganic crystalline materials. Examples are shown where, even when data for the target property are few, reusing the “features” of a model pretrained on related properties allows models to be built with only dozens of dataSourced [Ref. 3]
• Deep transfer learning by Jha et al. Combining a large DFT dataset (OQMD) with other smaller DFT datasets and experimental data, they report a mean absolute error of 0.07 eV/atom for predicting formation energy, using 1,643 experimental observations, which they say is comparable to the error of DFT itselfSourced [Ref. 4]
• Comparison by Hutchinson et al. Noting that a property with the same name may be measured or computed differently depending on the data source, they compare three transfer-learning approaches (multi-task, difference and explicit latent variable architectures)Sourced [Ref. 5]. It relates to the idea of a “source column” in the previous article
• The “crude estimation” of Zhang and Ling. They propose that adding a crude estimation of the property to the descriptors increases prediction accuracy without raising the degree of freedom, and report improved predictive capability in three case studies: the band gap of binary semiconductors, lattice thermal conductivity, and elastic properties of zeolitesSourced [Ref. 1]

We check the mechanism with a small experiment (Our calculation). The target property has only 15 measurements. We add, as one new descriptor, the prediction of a model pretrained on a large related dataset (3,000 points), and correct with the 15 points. For comparison we also try pretraining on data from a different, “unrelated” function (3,000 points).

The results are in Fig. 6. Training on the 15 measurements alone gives an error on new data (RMSE) of 0.67. Using the prediction of a model pretrained on related data brings it down to 0.05. But using a model pretrained on unrelated data makes the error 1.35, worse than with the measurements alone. A do-nothing prediction (the mean) gives 1.53. With 6 measurements, own data alone gives 1.33 and the related source 0.10; with 40, 0.44 and 0.03.

Our calculation. Synthetic data. The measured property was built from a function with different coefficients and terms than the related data (0.9 × the related function + 0.4 × descriptor 1). The related-data model is a regression on random features (3,000 points). Mean of 100 runs. This is an ideal case where the relation was built to be strong; it does not show the size of the effect for real materials.

8. Data augmentation: what to check before adding data

Data augmentation increases data by applying physically plausible changes to the original samples. For images it means rotation or added noise, but for materials the change must not alter the property. Oviedo and colleagues, in classifying crystallographic dimensionality and space group from thin-film X-ray diffraction patterns (a measurement that probes how crystals are arranged), compensated for scarce experimental data (115 thin films) with physics-informed data augmentation. Using simulated diffraction patterns and augmenting the experimental data as well, they report cross-validated accuracies of 93% for dimensionality and 89% for space groupSourced [Ref. 6].

Three cautions apply when using augmentation (commentary).

• Use only changes that leave the label unchanged. A small fluctuation that mimics measurement noise does not change the property, but arbitrarily altering a composition value changes it
• Split before augmenting, or split by original sample. As in Chapter 5, if copies fall in both training and evaluation, the evaluation looks better only in appearance (R² 1.00 versus −0.31)
• Do not skip confirmation on real data. A score on augmented data is not a score on real data. In the end, evaluate on samples that were actually measured (not augmented)

9. What to try, in order, with small data

Here is a summary, in order, of what to try when data are few (Fig. 7; our own organization).

1. Data quality. Check scatter, units, missing values and records of failure (our explainer on materials data and descriptors). The quality of a single point matters most
2. Simple baselines. Compare with “the mean of all data,” “the value of the closest past sample” and “linear regression.” If a complicated model cannot beat these, there is no point in using it
3. A few meaningful descriptors. Narrow the descriptors with physical and chemical knowledge (the crude estimation of Zhang and Ling [Ref. 1]; SISSO by Ouyang et al. is reported to give stable results even with small training sets [Ref. 7]Sourced)
4. Regularization. Rein in the descriptors with ridge or lasso (Chapter 6)
5. Related data. Try transfer learning and descriptors that carry crude estimates (Chapter 7). If the relation is weak, it backfires
6. Careful evaluation. Use repeated cross-validation, nested cross-validation and group-wise splits (Chapter 4). Do preprocessing, descriptor selection and augmentation only inside the training data (Chapter 5)
7. Tests of extrapolation. If the aim is to search for new materials, a number from a random split is overstated. Meredig and colleagues state that traditional machine-learning metrics overestimate model performance for materials discoverySourced [Ref. 17]. Xiong and colleagues proposed a k-fold-m-step forward cross-validation (sort the data, for example by value, and use the range ahead for evaluation) and showed that, even when conventional cross-validation looks good, the explorative prediction power is actually very lowSourced [Ref. 18]
8. Check by experiment. A prediction is a candidate, and confirmation is by experiment. Letting the model choose the next sample to measure (active learning) grows data efficiently. Li and colleagues showed for several materials datasets that up to 95% of the data can be removed with little effect on in-distribution performance, and report that uncertainty-based active learning can construct much smaller but equally informative datasetsSourced [Ref. 19] (see our explainer on Bayesian optimization and active learning)

10. Limits, what we could not confirm, and a summary

(1) What we could not confirm

• Numerical experiments. The calculations in Chapters 2 to 8 use synthetic data. They do not show the degree of overfitting, the swing of cross-validation, or the size of the effect of transfer learning in real materials
• The line for “small data.” We could not confirm a single definition in primary sources of how many points count as small data (as the table shows, size depends on conditions)
• A rule of thumb for descriptors versus data. We could not confirm a generally valid rule such as “at most so many descriptors per data point”
• Other methods. Gaussian-process regression, random forests, Bayesian methods and physics-informed methods are not covered here (uncertainty is covered in the next article)
• General effect of data augmentation on materials. We introduced one case (classifying X-ray diffraction, Oviedo et al.); we did not confirm its effect for other materials and properties
• Text of standards. For ISO/IEC standards we only checked number, title and publication date
• An international standard specific to small-data machine learning. We could not confirm one (Chapter 11)

11. How to read the yardsticks and standards

How good machine learning with small data is gets measured by prediction error (MAE, RMSE, R²), but the fewer the data, the more it matters how far that number can be trusted. The yardsticks are as follows. (1) The error metric (what it measures and what it does not). (2) How the data are split (random, by group, or for extrapolation). (3) The swing of the number (the spread of cross-validation). (4) Whether there is leakage. (5) Common benchmarks such as Matbench. The table lists what we could confirm (checked 22 October 2026). We could not confirm an international standard specific to small-data machine learning.

The arrangement of the table is Our calculation (organization). ISO/IEC numbers, titles, dates and stages were checked on the official ISO pages (22 October 2026). “Voluntary use” for the NIST AI RMF is from the document's own wording. Standard texts are paywalled, so we checked only titles and dates and wrote no requirements.

Cautions in reading (for materials engineers)

• Read a number together with its swing. An “R² = 0.85” from a study with few data could be anywhere from 0.4 to 0.9 for the same method, depending on how the data were drawn. Trust studies that report the spread (standard deviation, range)
• A benchmark ranking differs from performance on your material. On small tasks, differences in rank can fall within the swing
• Check that the evaluation procedure is written in the report: the unit of splitting, the timing of preprocessing and descriptor selection, and the separation of tuning from evaluation. If they are not written, leakage is possible
• Most standards and guidelines are recommendations. The NIST AI RMF is meant for voluntary use. ISO/IEC standards generally bind only when cited in law or contracts (commentary). When something says “compliant,” check which requirement of which document is meant

12. Glossary

13. References (primary sources)

1. Zhang Y., Ling C. "A strategy to apply machine learning to small datasets in materials science", npj Comput. Mater. 4, 25 (2018), DOI 10.1038/s41524-018-0081-z (open access) — doi.org/10.1038/s41524-018-0081-z
2. Dunn A. et al. "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Comput. Mater. 6, 138 (2020), DOI 10.1038/s41524-020-00406-3 (open access) — doi.org/10.1038/s41524-020-00406-3
3. Yamada H. et al. "Predicting Materials Properties with Little Data Using Shotgun Transfer Learning", ACS Cent. Sci. 5, 1717–1730 (2019), DOI 10.1021/acscentsci.9b00804 — doi.org/10.1021/acscentsci.9b00804
4. Jha D. et al. "Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning", Nat. Commun. 10, 5316 (2019), DOI 10.1038/s41467-019-13297-w (open access) — doi.org/10.1038/s41467-019-13297-w
5. Hutchinson M. L. et al. "Overcoming data scarcity with transfer learning", arXiv:1711.05099 — arxiv.org/abs/1711.05099
6. Oviedo F. et al. "Fast and interpretable classification of small X-ray diffraction datasets using data augmentation and deep neural networks", npj Comput. Mater. 5, 60 (2019), DOI 10.1038/s41524-019-0196-x (open access) — doi.org/10.1038/s41524-019-0196-x
7. Ouyang R. et al. "SISSO: A compressed-sensing method for identifying the best low-dimensional descriptor in an immensity of offered candidates", Phys. Rev. Mater. 2, 083802 (2018), DOI 10.1103/PhysRevMaterials.2.083802 — doi.org/10.1103/PhysRevMaterials.2.083802
8. Hoerl A. E., Kennard R. W. "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics 12, 55–67 (1970), DOI 10.1080/00401706.1970.10488634 — doi.org/10.1080/00401706.1970.10488634
9. Tibshirani R. "Regression Shrinkage and Selection via the Lasso", J. R. Stat. Soc. B 58, 267–288 (1996), DOI 10.1111/j.2517-6161.1996.tb02080.x — doi.org/10.1111/j.2517-6161.1996.tb02080.x
10. scikit-learn User Guide "Linear Models" (ridge regression, lasso) — scikit-learn.org/stable/modules/linear_model.html
11. scikit-learn User Guide "Cross-validation: evaluating estimator performance" — scikit-learn.org/stable/modules/cross_validation.html
12. Varoquaux G. "Cross-validation failure: Small sample sizes lead to large error bars", NeuroImage 180, 68–77 (2018), DOI 10.1016/j.neuroimage.2017.06.061 — doi.org/10.1016/j.neuroimage.2017.06.061
13. Cawley G. C., Talbot N. L. C. "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", J. Mach. Learn. Res. 11, 2079–2107 (2010) — jmlr.org/papers/v11/cawley10a.html
14. Varma S., Simon R. "Bias in error estimation when using cross-validation for model selection", BMC Bioinformatics 7, 91 (2006), DOI 10.1186/1471-2105-7-91 — doi.org/10.1186/1471-2105-7-91
15. Ambroise C., McLachlan G. J. "Selection bias in gene extraction on the basis of microarray gene-expression data", Proc. Natl. Acad. Sci. USA 99, 6562–6566 (2002), DOI 10.1073/pnas.102102699 — doi.org/10.1073/pnas.102102699
16. Kapoor S., Narayanan A. "Leakage and the reproducibility crisis in machine-learning-based science", Patterns 4, 100804 (2023), DOI 10.1016/j.patter.2023.100804 (open access) — doi.org/10.1016/j.patter.2023.100804
17. Meredig B. et al. "Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery", Mol. Syst. Des. Eng. 3, 819–825 (2018), DOI 10.1039/C8ME00012C — doi.org/10.1039/C8ME00012C
18. Xiong Z. et al. "Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation", Comput. Mater. Sci. 171, 109203 (2020), DOI 10.1016/j.commatsci.2019.109203 — doi.org/10.1016/j.commatsci.2019.109203
19. Li K. et al. "Exploiting redundancy in large materials datasets for efficient machine learning with less data", Nat. Commun. 14, 7283 (2023), DOI 10.1038/s41467-023-42992-y (open access) — doi.org/10.1038/s41467-023-42992-y
20. Kirklin S. et al. "The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies", npj Comput. Mater. 1, 15010 (2015), DOI 10.1038/npjcompumats.2015.10 (open access) — doi.org/10.1038/npjcompumats.2015.10
21. ISO/IEC "ISO/IEC TS 4213:2022 Information technology — Artificial intelligence — Assessment of machine learning classification performance" (ISO page) — iso.org/standard/79799.html
22. ISO/IEC "ISO/IEC 23053:2022 Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)" (ISO page) — iso.org/standard/74438.html
23. ISO/IEC "ISO/IEC 5259-1:2024 Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 1: Overview, terminology, and examples" (ISO page) — iso.org/standard/81088.html
24. ISO/IEC "ISO/IEC 5259-3:2024 Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 3: Data quality management requirements and guidelines" (ISO page) — iso.org/standard/81092.html
25. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1, January 2023, DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1

14. Claim-to-source audit

Last updated: 22 October 2026 / Sources are limited to primary materials (peer-reviewed papers, arXiv papers, and official pages of scikit-learn, ISO and NIST). The full text of ISO/IEC standards is paywalled and was not checked. The numerical experiments in Chapters 2 to 8 are calculations on synthetic data we generated, not results on real materials. We could not confirm a single definition of “small data,” a general rule of thumb for descriptors versus data, the general effect of augmentation on materials, or an international standard specific to small-data machine learning, so these are not stated. All figures are explanatory concept diagrams or drawings that include our own calculation; the hero image and Fig. 8 are ImageGen-generated conceptual illustrations.