MENU
Advanced Materials Intelligence
  1. Home
  2. Large Language Models and Text Mining for Materials Explained

Large Language Models and Text Mining for Materials Explained

← Back to Materials Informatics guides

TECHNOLOGY EXPLAINER

Large Language Models and Text Mining for Materials
— pulling materials data out of papers and patents

Much of what is known about materials sits in the text of papers and patents; the tables in databases hold only a small part of it. This article explains, step by step and with figures and small calculations, how data can be pulled out of text automatically (text mining). It follows the path from dictionaries and rules, through BERT-type models, to large language models (LLMs), and looks at how accurate the extraction really is, where LLMs go wrong (hallucination, mixed-up numbers and units) and what models exist for materials. The stance is not "ask an LLM and get the answer" but "an LLM is a strong tool for turning text into tables, and its output must be checked."

Built from papers by Kononova et al., Tshitoyan et al., Dagdelen et al., Polak and Morgan and Zheng et al.; the MaScQA, ChemBench and MatSci-NLP evaluation papers; and ISO/IEC and NIST pages. Worked numbers are our own calculations. Checked 22 October 2026

Extract material names, conditions and values, then check the original source. An LLM-filled table still requires checks of units and context.
Fig. 1. Conceptual illustration generated with ImageGen. Extract material names, conditions and values, then check the original source. An LLM-filled table still requires checks of units and context.Not an actual product, facility or measurement. Enlarged models, lines, arrows and colours are explanatory.
What this article covers
  1. The short version / what to know first / place in the series
  2. Why pull data out of text at all
  3. From text to a table: six steps
  4. A real case: 4.2 million papers to solid-state reactions
  5. How methods evolved: from rules to LLMs
  6. How accurate is LLM extraction?
  7. Where LLMs fit and where they fail: hallucination, numbers, tables and figures
  8. LLMs and foundation models for materials: what the papers show
  9. A materials engineer's view: starting with your own documents
  10. Limits, what could not be confirmed, and a summary
  11. How to read the yardsticks and standards
  12. Glossary
  13. References
  14. Claim-to-source audit
How claims are labelled in this article

Sourced = stated in published material (link given)
Our calculation = a figure this article derived, with the assumptions spelled out
Not yet confirmed = a plan, target, draft or other item that is not settled
Interpretations for materials engineers are marked separately as commentary; inference is shown in bold; claims by research groups or companies are labelled as claims.

How to read the numbers in this article

Precision, recall and accuracy here are values for that paper's task, data and way of scoring. Please do not set numbers from different papers side by side and call one "better". They describe how well text was read, and do not guarantee that an extracted material value is correct in an experiment.

1. The short version / what to know first / place in the series

  • Much materials data lives in the text of papers and patents. Turning it into a table is text mining. In a landmark case, about 19,000 solid-state synthesis reactions were extracted from about 4.2 million papersSourced[Ref. 1]
  • LLMs are strong at "rewriting text into a fixed table." In published tests precision and recall both reached about 90%. But when the definition of "correct" changed, recall fell to about 63% in one case, and hallucination (adding content not in the text) and mixed-up numbers or units also occurSourced[Ref. 11, 10]
  • The one thing to remember: an extracted value is a candidate. Checking against the source, and measuring accuracy on an answer set made by people, must be built into the processSourced[Ref. 11, 20]
What to know before reading

Machine learning is "an approximator that learns trends from past data", and an LLM is "a model that writes text by predicting the next word" (see the series explainers on machine-learning basics and on Transformers and LLMs). They are not repeated here; this article concentrates on reading text and building tables. It can be read alone, and terms are explained where they first appear.

Place in the series

The shape of data is covered in the explainers on materials data and descriptors and on materials databases and FAIR; LLMs using tools such as calculations and instruments are in the one on AI agents for materials research; LLMs proposing materials themselves are in the one on generative AI for materials design; and judging results is in the one on evaluation and reproducibility. This article is the entry point: making raw material for materials data out of large amounts of text.

2. Why pull data out of text at all

The results of materials experiments, successful or not, first appear as sentences in papers: for example, "this material formed when these powders were fired at this temperature for this long." Reading them one by one would help, but there are far too many.

A quick calculation gives a feel for the scale. One study (Kononova et al., 2019) collected 4,204,170 papers[Ref. 1]. Assuming 10 minutes per paper, that is about 700,000 hours. If one person reads 2,000 hours a year (8 hours a day for 250 days), it comes to about 350 person-yearsOur calculation (the 10 minutes is an assumption chosen only to show the order of magnitude).

Tshitoyan et al. note that structured materials databases hold only a small fraction of the knowledge in the literatureSourced[Ref. 3]. If materials discovery is "searching a vast map for buried treasure", past papers are the logs left by earlier explorers, and text mining is the work of copying those logs onto the map. A copying error makes the map itself wrong.

Where the analogy breaks

The explorers' logs are biased. Papers carry many successes, while unsuccessful results ("no" results) are rarely publishedSourced[Ref. 2]. A map copied faithfully from them shows mostly the roads where treasure was found. This bias is a separate problem from the accuracy of text mining.

Easily confused: what is "text mining"?

It is the general name for reading text by computer, finding information such as material names and numbers, and putting it into a table. The wider field of handling text by computer is called natural language processing (NLP). For this article it is enough to remember "text to table".

3. From text to a table: six steps

Turning text into a table can be broken into about six steps, and each has its own ways to go wrong. The flow is the same when an LLM is used.

1 source document, 2 extraction region, 3 structured fields, 4 checking. LLM output should not be registered as measured data without validation.
Fig. 2. Conceptual illustration generated with ImageGen. 1 source document, 2 extraction region, 3 structured fields, 4 checking. LLM output should not be registered as measured data without validation.Not an actual product, facility or measurement. Enlarged models, lines, arrows and colours are explanatory.
View original SVG, values and sources
From the text of a paper to a usable table: six steps and where each can go wrongStepWhat is doneWhat can go wrong11 CollectGet the full text of papers and patents. HTML/XML iseasy to handle; PDF mixes columns, tables andequations.Bulk download needs atext-and-data-mining (TDM) agreement.Recovering PDF structure scores belowabout 80% F1.22 Select paragraphsPick out only the paragraphs you need, such assynthesis or testing (classification).Unrelated paragraphs raise errors;missed ones leave gaps in the data.33 Find entitiesMark material names, temperatures, times and values inthe text (named entity recognition).Abbreviations, alternative names andformula styles vary; similarproperties (e.g. bulk and Young'smodulus) get mixed up.44 Link themConnect the marked words: "this temperature belongs tothe firing of this material" (relation extraction).A sentence with several materials andvalues is easy to mismatch.55 Make a tablePut values and units into a fixed layout (a schema,e.g. JSON).Ranges (10-100) and inequality signs(<0.1) get dropped; unitconversions go wrong.66 CheckCompare with the source and measure precision andrecall against answers made by people.Without this step, errors stayunnoticed in the database.* Conceptual diagram. The names and split of steps are for explanation; not every study follows exactly these six steps. F1 combines precisionand recall.

Original SVG reference: Conceptual diagram (vector drawing).Six steps for building a table from text. The PDF and TDM remarks come from the review by Kononova et al. (2021) and the 2019 dataset paper.

The core is steps 3 and 4. Step 3 is named entity recognition (NER): marking words as "material name", "temperature" and so on. Step 4 is relation extraction: connecting them, as in "this temperature belongs to the firing of this material". With step 3 alone you have temperatures scattered around with no way to tell whose they are.

Here is a small example. The left side of the figure is a sentence written for this article for illustration (not a quotation from a real paper); the right side is the extracted and organized result.

Example: one synthesis paragraph becomes a structured "recipe" (illustrative)Text of a paper (an example written for this article)Extracted result (a fixed layout)BaCO₃ and TiO₂ were weighed in equimolar amountsPrecursorsBaCO₃, TiO₂ (mole ratio 1:1)and ball-milled in ethanol.Step 1: mixingball mill; medium: ethanolAfter drying, the powder was calcined in air at 1100°C for 4 hStep 2: drying; Step 3: calciningcalcining: 1100 °C, 4 h, in airto give BaTiO₃.TargetBaTiO₃Balanced reactionBaCO₃ + TiO₂ → BaTiO₃ + CO₂* Numbers are illustrative. This is not a quotation from areal paper.
Fig. 3 Conceptual diagram (vector drawing).An example of rewriting prose into materials, steps and conditions. In the dataset of Kononova et al. (2019) each record holds the target, precursors, operations with conditions and a balanced reaction.

Once the reaction is balanced, as at the bottom right, you can later count which precursors and temperatures gave which material. Each record in the dataset of Kononova et al. likewise holds a target, precursors, operations with conditions and a balanced reactionSourced[Ref. 1].

The entrance problem: getting the text

  • HTML/XML full text is easy to handle, but PDF mixes columns with tables and equations and makes it hard to recover the reading order. The review reports that methods for restoring the logical structure of PDFs still score below about 80% F1Sourced[Ref. 2]
  • Downloading large amounts of full text needs a text and data mining (TDM) agreement with each publisher. Kononova et al. obtained permission from six publishersSourced[Ref. 1, 2]. Checking rights and licences is a separate task from the technology (this article is not legal advice)
  • Patents can be treated the same way. For example, a dataset of chemical reactions text-mined from US patents published between 1976 and September 2016 is public (CC0). Its notes warn that duplicates are frequent, and it holds chemical reactions written as SMILES and similar, not a collection of material compositions or propertiesSourced[Ref. 22]

4. A real case: 4.2 million papers to solid-state reactions

In 2019 Kononova et al. released a dataset of inorganic "synthesis recipes" pulled from papersSourced[Ref. 1]. Solid-state synthesis means mixing powder precursors and firing them at high temperature to obtain the target material. The figure below shows how the numbers shrink from step to step.

A real case: 4.2 million papers to about 19,000 solid-state reactions (Kononova et al. 2019)Counts (log-scale bars; length is not proportional to the count)Papers scraped4,204,170Paragraphs in experimental sections6,218,136about 1.5 per paperInorganic synthesis paragraphs188,198about 3.0% of those paragraphsof which solid-state53,538about 28.4% of synthesis paragraphsParagraphs giving a balanced reaction15,144about 28.3% of solid-state paragraphs (paper: 28%)The final dataset has 19,488 reactions; it exceeds the 15,144 paragraphs because oneparagraph can give several reactions.Accuracy of the extracted records(100 random paragraphs checked by hand)Chemistry level (precursors, target,reaction)93%Chemistry plus operations, withoutheating/mixing conditions64%All items including operation conditions51%Of the 100 paragraphs, 30 did not state allprecursors and products, so even an expertcould not rebuild a reaction.* Counts and accuracy are as stated in the paper; the percentages between steps are our calculation. Log-scale bars understate differences incount.
Fig. 4 Figure including our calculation (vector drawing).Only about 28% of solid-state paragraphs yielded a balanced reaction. The authors say they chose precision over recall: better to miss a record than to give a wrong one.
  • There were 4,204,170 papers, 6,218,136 paragraphs in experimental sections, 188,198 inorganic-synthesis paragraphs and 53,538 solid-state paragraphsSourced[Ref. 1]. A balanced reaction could be built from 15,144 of the solid-state paragraphs, about 28%Our calculation (the paper states 28%)
  • The authors chose to miss records rather than output wrong ones (high precision, low recall)Sourced[Ref. 1]. In a check of 100 paragraphs, 93% were right at the chemistry level and 51% once every operation condition was included
  • As another case, a battery-materials database was built automatically with ChemDataExtractor, giving 292,313 records from 229,061 papersSourced[Ref. 6]
Easily confused: why 93% and 51%?

The same records look more or less correct depending on how much you check. For "precursors, target and reaction" alone it is 93%; once every detail such as mixing equipment and medium must match, it drops to 51%. The authors note that the details of mixing are often unimportant in solid-state synthesis. When you see a figure, check which items it covers.

A materials engineer's view 1: an extracted "recipe" is a conditional report

A recipe in a paper is a report made with that lab's precursor purity, particle size, furnace and atmosphere. The "1100 °C, 4 h" in an extracted table will not necessarily reproduce with your powder and your furnace (commentary). Use the database as a tool for narrowing where to look, and confirm in your own lab with experiments that include controls and repeats (including lot-to-lot variation).

5. How methods evolved: from rules to LLMs

The way extraction is done has changed over time. A 2021 review described text mining in materials science as still at an early stageSourced[Ref. 2]. We look at it in three broad generations.

How extraction methods evolved: dictionaries and rules, then BERT-type models, then LLMs1 Dictionaries, rules and small MLHow: find names and numbers withchemical dictionaries and patternsExample: ChemDataExtractor (2016)Strength: transparent and fastWeakness: breaks when wording changes;people write the rules2 BERT-type models (fine-tuned)How: a model pretrained on scientifictext is fine-tuned on examples marked bypeopleExamples: SciBERT, MatBERT, MatSciBERTStrength: high accuracy on a definedtaskWeakness: needs many annotated examplesper task3 Large language models (LLMs)How: ask with a prompt, or fine-tune onroughly 100-500 examplesExamples: ChatExtract, LLM-NERRE,ChemPromptStrength: handles complex relations withfew annotationsWeakness: hallucination, number mix-ups,run-to-run variationRough timelinerule-based → pretrained models → instruction-following large models* Conceptual diagram. The three are not strictly sequential; each is still used depending on the task. All LLM examples have been tested onmaterials literature in a paper.
Fig. 5 Conceptual diagram (vector drawing).Comparison of the three approaches. The advantage of domain pretraining is from Trewartha et al.; the annotation amount for LLMs is from Dagdelen et al.

1. Dictionaries, rules and small machine learning

ChemDataExtractor is a toolkit for chemical text that finds material names and numbers with rules and dictionariesSourced[Ref. 4]. Weston et al. built a named-entity model that marks materials, sample descriptors, phases, properties, applications and synthesis methods; it reached an F1 of 87% and was applied to 3.27 million abstracts, extracting more than 80 million entitiesSourced[Ref. 5].

2. Word vectors and BERT-type models

Tshitoyan et al. turned words into lists of numbers (vectors), called word embeddings, using paper abstracts. Without any labels from people, the vectors reflected the structure of the periodic table and structure-property relations, and the authors report recommending candidate functional materials several years before their discoverySourced[Ref. 3]. This tests past discoveries after the fact and does not guarantee future ones (commentary).

Next came BERT-type models. BERT is a Transformer model that reads text and turns each word into numbers. There are SciBERT, pretrained on scientific papers[Ref. 7], and MatBERT and MatSciBERT, pretrained on materials papers. Trewartha et al. compared them on three materials named-entity datasets and found MatBERT 1-12% better than the other BERT-base models (BERT and SciBERT), and the margin of MatBERT and SciBERT over the original BERT larger when data are scarceSourced[Ref. 8]. They also report that a simple BiLSTM using materials-specific word vectors consistently beat the original BERT. A newer or bigger model is not always better.

3. Large language models (LLMs)

An LLM works when you simply ask with an instruction (a prompt) such as "pull the materials and values out of this text". If you also fine-tune it (train it a little more) on roughly 100-500 examples made by people, it can write complex relations in a fixed format. Dagdelen et al. fine-tuned GPT-3 and Llama-2 on three tasks, including linking dopants to host materials and cataloguing MOFsSourced[Ref. 10]. The next chapter looks at accuracy.

Easily confused: pretraining and fine-tuning

Pretraining is learning how words are used from a huge amount of text in advance, like basic training for new employees. Fine-tuning is extra training on a few examples in your own format, like on-the-job training after assignment. Where the analogy breaks: an LLM does not "understand" what it learned, it picks up statistical patterns of text.

6. How accurate is LLM extraction?

Two numbers are needed to talk about accuracy.

  • Precision: the share of extracted values that were right. How trustworthy the answers given are
  • Recall: the share of the true values that were found. How few were missed
Easily confused: think of a fishing net

Picture catching a particular fish in a pond. Precision is "of the fish in the net, the share that are the right kind"; recall is "of the right fish in the pond, the share that were caught". A coarse net keeps mostly the right fish (high precision) but lets small ones slip away (low recall). The approach of Kononova et al. is close to this coarse net. Where the analogy breaks: in data extraction, what counts as the "right fish" is decided by an answer set made by people.

More detail: a worked example of precision, recall and F1

Suppose a document set holds 100 true values, and the system extracts 80, of which 72 are right. Precision = 72/80 = 90%; recall = 72/100 = 72%. F1 combines the two: F1 = 2 × precision × recall / (precision + recall) = 2 × 0.90 × 0.72 / 1.62 = 0.80Our calculation (illustrative numbers).

Now the published numbers.

One method, different scores: task and definition of "correct" matter (reported values)0%25%50%75%100%90.887.7Bulk modulus(GPT-4, 100 sentences)76.963.1Critical cooling rate(all mentions)91.984.2Critical cooling rate(standardized)Precision (share of extracted values that are right)Recall (share of true values that were found)How to readLeft: a test set of 100 sentences (179 data points)gives about 90%.Middle: counting every mention as correct brings manyranges and material families, and recall falls toabout 63%.Right: restricted to uniquely identified materials andvalues, it returns to about 92% and 84%.The same method on the same papers can score 60% or90%, depending on what counts as correct.* Values in the body of Polak and Morgan (2024), one decimal (their abstract gives 91.6% and 83.6% for the right-hand case). The three settingsdiffer; the chart shows the effect of defining "correct", not a ranking. Do not compare with other studies (e.g. 90-99% by Zheng et al.).
Fig. 6 Plot of reported values (vector drawing).Plot of reported ChatExtract results. For critical cooling rate the reference is a carefully hand-made database (721 entries); ChatExtract returned 634 values.
  • ChatExtract (Polak and Morgan): a conversational LLM is made to judge whether a sentence holds data, extract the value and answer follow-up checks. On a bulk-modulus test (100 sentences, 179 data points) GPT-4 scored 90.8% precision and 87.7% recall. On a real database build for the critical cooling rate of metallic glasses, the standardized version scored 91.9% and 84.2% (the abstract and introduction give 91.6% and 83.6%, slightly different from the body text; this article uses the body values)Sourced[Ref. 11]
  • The same task, a different definition of "correct": for critical cooling rate, counting every numerical mention as correct gave 76.9% precision and 63.1% recall, because there were many ranges and material families (such as Zr-based glasses), and a single dropped symbol counted as an errorSourced[Ref. 11]
  • LLM-NERRE (Dagdelen et al.): accuracy drops for complex relations. F1 on the MOF and general materials tasks was about 30% lower than on the dopant-host task, and the best was 0.537 for formula-application links (an exact word-match criterion, so a lower bound)Sourced[Ref. 10]
  • ChemPrompt (Zheng et al.): 26,257 synthesis parameters were extracted from papers on about 800 MOFs, with precision, recall and F1 of 90-99% reported. A machine-learning model predicting crystallization outcomes (over 86% accuracy) was then built from the tableSourced[Ref. 12]

Here is what "about 90%" means. At 90% precision, if 1,000 values are extracted, about 100 are wrong. At 84% recall, about 16% of the values in the papers never reach the tableOur calculation (simple arithmetic on the numbers above). And you cannot tell which 100 are wrong, which is why a way back to the source and spot checks are needed.

Common misconception: "give the papers to an LLM and a correct table comes out on its own"

This is not correct. Even the best published results sit around 90% precision, and depending on the task and definition of "correct" recall falls to the 60s. The accurate statement is: "an LLM drafts the table quickly; it becomes a usable table only with fields defined by people, measurement on an answer set and checking against the source."

7. Where LLMs fit and where they fail: hallucination, numbers, tables and figures

Where an LLM fits, where to verify, and where not to rely on it (our summary)Good fit: turning text into afixed layout- Pick up materials, conditions andvalues into a table (fix the fieldsfirst)- Classify paragraphs (synthesis ortesting)- Draft a mapping of abbreviations tofull names- Draft summaries and plain-languagerewritesVerify: where it slips- Numbers and units (similar properties,ranges, inequality signs)- Which sample a value belongs to- Adding names or formulae that are notin the text (hallucination)- Information inside tables, figures andPDFsDo not rely on: answers frommemory- Asking for property values without asource- Using a guessed property of an unknownmaterial as it is- Using it in place of calculation orexperiment- Guessing conditions that the paperdoes not stateEvidence (from the papers)- Dagdelen et al. saw hallucinations such as adding the formula SiO₂ where the text only says "silica".- Polak and Morgan: confusion between similar properties (bulk and Young's modulus) is a key difficulty.- Zaki et al. (MaScQA): GPT-4 about 62% correct; about 64% of errors conceptual, about 36% computational.- Mirza et al. (ChemBench): better than chemists on average, yet wrong on some basic tasks and overconfident.
Fig. 7 Conceptual diagram (vector drawing).The colour coding is our own summary (commentary); the evidence is in the papers listed in the lower box.

(1) Hallucination

Hallucination means writing something plausible that is not in the source. The NIST generative AI profile lists confabulation, "the production of confidently stated but erroneous or false content" (colloquially hallucination), as one of the risks of generative AISourced[Ref. 29]. A materials example is reported by Dagdelen et al.: the text says only "silica", and the model writes "SiO₂" in the formula field. It may be right, but because it is not in the text it should not be entered as data, the authors saySourced[Ref. 10].

(2) Numbers, units and "whose value is it?"

Polak and Morgan explain that the bulk-modulus task is hard because papers also report Young's modulus, shear modulus and others, with similar names, value ranges and units (pressure). In their test set 70% of the sentences contained several valuesSourced[Ref. 11]. A mix-up of which value belongs to which material looks plausible and is hard to spot (commentary).

On the materials question set MaScQA (650 questions), GPT-4 answered about 62% correctly, and about 64% of the errors were conceptual and about 36% computationalSourced[Ref. 19]. On the chemistry benchmark ChemBench (over 2,700 questions), the best models on average beat the best human chemists but erred on some basic tasks and gave overconfident predictionsSourced[Ref. 20].

(3) Tables, figures, PDFs and what is never published

Information in tables and figures needs a different technology (such as image recognition) from text extractionSourced[Ref. 2]. The review also names the bias of papers toward successes as a problem. Raccuglia et al. showed that using failed experiments recorded in lab notebooks in machine learning helped predict the conditions under which crystals formSourced[Ref. 21].

(4) Ways to reduce hallucination and mix-ups

The ChatExtract authors list measures such as (a) stating explicitly that the data may be missing so the model is not pushed to fill gaps, (b) adding follow-up questions that invite a negative answer so it reconsiders, and (c) asking all questions within one conversation and showing the full data each timeSourced[Ref. 11]. All follow one idea: check the same answer by asking it another way.

A materials engineer's view 2: record the version and the date

LLMs are updated, and the same prompt can give different results (commentary). If you use extraction in real work, record the model name and version, the run date, the prompt and the settings together with the data. Being able to trace why a value entered the table matters especially for materials data. To reproduce results you also need to keep the source sentence.

8. LLMs and foundation models for materials: what the papers show

A "foundation model" is one pretrained on a large amount of data and reused for many tasks. The table lists models for materials text as far as the papers allow. Tasks, data and evaluation conditions differ between papers, so the models are not ranked against each other.

Model or methodKindWhat the paper states (as checked)Note
SciBERT (2019)BERT-typePretrained on multi-field scientific papers; reported better than BERT on several tasksSourced[Ref. 7]Not specific to materials
MatBERT (2022)BERT-typePretrained on materials papers; 1-12% better than the other BERT-base models on three named-entity datasetsSourced[Ref. 8]A BiLSTM with materials word vectors also beat BERT in the same study
MatSciBERT (2022)BERT-typePretrained on materials papers; reported to beat SciBERT on abstract classification, named-entity recognition and relation extraction; weights releasedSourced[Ref. 9]Results on the paper's evaluation sets
DARWIN (2023)Fine-tuned LLMAn open-source LLM fine-tuned on over 60,000 instruction examples in physics, chemistry and materialsSourced[Ref. 15]Preprint; evaluation is on the paper's tasks
LLaMat (2024)LLM (continued pretraining)A family made by continued pretraining of LLaMA on materials literature and crystal data; the authors say the Llama-2-based version unexpectedly performed better on materials tasks than the Llama-3-based oneSourced[Ref. 14]Preprint; an example where the newer base model was not better
LLM-Prop (2025)LLM for property predictionPredicts band gap and other properties from text descriptions of crystals; the authors report beating a GNN-based method on their TextEdge dataSourced[Ref. 16]Comparison on the authors' own data
Fine-tuned LLaMA-2 (Gruver et al. 2024)LLM generating crystalsWrites crystal structures as text; 49% were predicted metastable by calculation (28% for the diffusion model CDVAE)Sourced[Ref. 17]A calculated prediction, not a synthesized result. See the explainer on generative AI for materials design

On the evaluation side there are MatSci-NLP, a collection of seven tasks on materials textSourced[Ref. 18], MaScQA, a question set on materials knowledge[Ref. 19], and ChemBench, which compares chemical knowledge with that of chemists[Ref. 20]. A hackathon report (Jablonka et al.) says working prototypes for property prediction, tool interfaces, knowledge extraction and education were built in under two daysSourced[Ref. 13]. That says prototypes are easy to make; it is not a guarantee of accuracy.

Common misconception: "a materials-specific LLM is always more accurate"

Not necessarily. Pretraining on materials papers has a measured advantage on tasks such as named-entity recognitionSourced[Ref. 8, 9]. But the LLaMat paper itself says that the version built on the older Llama-2 unexpectedly did better on materials tasks than the one built on the newer Llama-3Sourced[Ref. 14]. Choose by measuring on your own task and data.

9. A materials engineer's view: starting with your own documents

The same thinking applies not only to papers but also to your own lab reports, test certificates, specifications and patents. The figure shows how to start.

A way to start extraction from your own documents and papers (our summary)1 Fix the purpose andfieldsWhich table do you need?Material, composition,conditions, value, unit,method, sample history.2 Make an answer setPeople build the table froma few paragraphs. Studieshave used roughly 100-500examples.3 Trial extractionFix the prompt and run iton the answer-setparagraphs.4 Measure accuracyLook at precision andrecall for each field, andfor units and ranges.5 Fix error typesStudy the kinds ofmistakes, revise the promptor definitions, and go backto 3.6 Run on everythingKeep the source sentenceand document ID next toevery extracted value.7 Spot-checkPeople compare a sample ofoutputs with the source andrecord the error rate.8 Use itUse values as candidates;confirm important onesagainst the source beforeadopting them.* The dashed arrow shows the loop back to step 3 using the mistakes found in step 5. The number and names of steps are for explanation; this isnot a standard procedure.
Fig. 8 Conceptual diagram (vector drawing).From preparation to use. The size of the answer set follows Dagdelen et al.; confirming with redundant prompts follows the idea of Polak and Morgan.
A materials engineer's view 3: decide the fields (the schema) first

Before the LLM, decide what the columns of the table are. If you separate material, composition, lot, processing history, test method, specimen shape, test conditions, value and unit, you can later compare only values measured under the same conditions. An LLM is good at filling columns, but which columns matter for your decisions is for the experts to decide (commentary).

A materials engineer's view 4: failure records are valuable data that papers lack

Papers are biased toward successesSourced[Ref. 2]. As Raccuglia et al. showed, failure records in lab notebooks help machine learningSourced[Ref. 21]. Applying the same extraction to your own notebooks and defect reports can yield data on "conditions that do not work", which papers do not give (commentary). Whether to put documents into an outside service should be decided only after checking confidentiality agreements and internal rules (this article is not legal advice).

Leave unreported conditions missing rather than inventing them. A published recipe is not automatically a safe executable procedure.
Fig. 9. Conceptual illustration generated with ImageGen. Leave unreported conditions missing rather than inventing them. A published recipe is not automatically a safe executable procedure.Not an actual product, facility or measurement. Enlarged models, lines, arrows and colours are explanatory.

10. Limits, what could not be confirmed, and a summary

(1) What could not be confirmed

  • Each figure holds only for that paper's task, data and scoring. Repeat tests with the latest LLMs were not checked here.
  • LLaMat and DARWIN are preprints; a peer-reviewed journal version was not found.
  • The licence of each model (including commercial use) was not checked here; confirm the official terms before use.
  • A primary source evaluating extraction of material properties or compositions from patents in the materials field was not found (only an organic-reaction dataset was confirmed).
  • The current TDM terms of individual publishers were not checked.
  • No international standard specific to data extraction from the materials literature was found.
  • ISO/IEC standard texts are paid, so only titles and scope statements are given; requirements are not described.
Summary
  • Much materials knowledge is in text, and text mining automates "text to table". One case extracted about 19,000 solid-state reactions from about 4.2 million papersSourced
  • LLMs are strong at drafting tables. In the best published tests precision and recall are both about 90%, yet they fall to about 60% when the definition of "correct" changesSourced
  • Hallucination, mixed-up numbers and units, tables and figures, and unpublished failures are the main limits. Keep the source text, measure against an answer set and spot-checkSourced
  • A materials-specific, newer or bigger model is not always better. Choose by measuring on your own taskSourced
  • Decide the fields (schema) first, record version, date and prompt, and include your own failure records (commentary)

11. How to read the yardsticks and standards

The first yardstick here is correctness of extraction: precision, recall and F1 measured against an answer set made by people. Then there are question sets that test the materials and chemistry knowledge of LLMs (MaScQA, ChemBench and others). The former measures reading skill and the latter stored knowledge, and neither measures whether an experimental value for your own material is correct. Checked on 22 October 2026.

What is assessedRepresentative benchmark or method (name, source)Standards and guidance (ISO/IEC, NIST, international principles)How to read it
Extraction correctnessPrecision, recall and F1. ChatExtract used an answer set of 100 sentences and 179 data points[Ref. 11]. MatSci-NLP has seven tasks[Ref. 18]SourcedNot found (no international standard specific to materials text extraction was found)The value depends on what counts as correct (76.9%/63.1% versus 91.9%/84.2% for the same method); exact match is a lower bound[Ref. 11, 10]
Materials and chemistry knowledgeMaScQA (650 questions)[Ref. 19], ChemBench (over 2,700 questions)[Ref. 20]SourcedNot foundA test score differs from accuracy on your extraction task; look at the type of error (conceptual or computational) and at overconfidence
Data qualitySpot checks by people (for example, 100 paragraphs checked)[Ref. 1]SourcedISO/IEC 5259-1:2024 (data quality for analytics and machine learning, Part 1: overview, terminology and examples)[Ref. 26]SourcedA framework of concepts and terms, not a pass/fail criterion for materials data (text is paid; only the scope was checked)
Sharing and reuse of dataNot foundFAIR Guiding Principles (Wilkinson et al., 2016)[Ref. 23]SourcedPrinciples, not a legally binding standard; rights to extracted data depend on the licence of the source papers
AI terms, framework and managementNot foundISO/IEC 22989:2022 (concepts and terminology)[Ref. 24], ISO/IEC 23053:2022 (framework for AI systems using machine learning)[Ref. 25], ISO/IEC 42001:2023 (AI management system)[Ref. 27]SourcedThese cover terminology and how organizations run AI. 42001 specifies requirements for an AI management system; it does not guarantee that extracted values are correct
Generative AI risk (hallucination)Not foundNIST AI RMF 1.0 (NIST AI 100-1, January 2023)[Ref. 28] and NIST AI 600-1 Generative AI Profile (26 July 2024; lists confabulation among the main risks)[Ref. 29]SourcedVoluntary guidance, not a legal obligation

The arrangement of the table is Our calculation (organization). Numbers, titles and status checked on the ISO and NIST pages (22 October 2026). Standard texts are paid, so only the scope is described and requirements are not.

Reading notes (for materials engineers)

  • A benchmark score is not accuracy on your extraction task: a high MaScQA or ChemBench score does not mean the same accuracy on the notation and abbreviations in your reports. Build an answer set from your own documents and measure
  • "Extracted" and "verified" are different: a value pulled from text is a report in a paper, not a value confirmed by experiment. Do not blur the two
  • Separate recommendation from requirement: FAIR and the NIST guidance are voluntary, and ISO/IEC 42001 sets requirements for organizations that seek certification. None sets an accuracy criterion for extraction

12. Glossary

Text mining
Reading text by computer, finding information and putting it into a table.
Natural language processing (NLP)
The general name for handling human language by computer.
Named entity recognition (NER)
Marking words in text as material name, temperature and so on.
Relation extraction
Linking found words, for example which value belongs to which material.
Schema
A blueprint fixing the columns or fields of a table.
JSON
A data format that writes fields and values in nested form.
Corpus
A collection of text used for training or analysis.
Annotation
People marking correct answers on text.
Answer set (ground truth)
A table of correct answers made by people; the basis for measuring accuracy.
Pretraining
The stage of learning how words are used from a large amount of text.
Fine-tuning
Extra training on a few examples for a specific task.
BERT
A smaller Transformer model that reads text and turns each word into numbers.
Word embedding
A word turned into a list of numbers (a vector).
Large language model (LLM)
A large model that writes text by predicting the next word.
Foundation model
A model pretrained on large data and reused for many tasks.
Prompt
The instruction given to an LLM.
Hallucination (confabulation)
Writing plausible content that is absent from the source or wrong.
Precision
The share of extracted values that were right.
Recall
The share of true values that were found.
F1
A single score combining precision and recall (their harmonic mean).
Exact match
Strict scoring that counts only identical words; paraphrases are errors.
TDM (text and data mining)
Processing large amounts of paper text by machine; a publisher agreement is often needed.
Solid-state synthesis
Mixing powder precursors and firing at high temperature to obtain the target material.
MOF
Metal-organic framework: a porous crystal built from metals and organic molecules (see another article in this series).

13. References

  1. Kononova O. et al. "Text-mined dataset of inorganic materials synthesis recipes", Sci. Data 6, 203 (2019), DOI 10.1038/s41597-019-0224-1 — doi.org/10.1038/s41597-019-0224-1
  2. Kononova O. et al. "Opportunities and challenges of text mining in materials research", iScience 24, 102155 (2021), DOI 10.1016/j.isci.2021.102155 — doi.org/10.1016/j.isci.2021.102155
  3. Tshitoyan V. et al. "Unsupervised word embeddings capture latent knowledge from materials science literature", Nature 571, 95–98 (2019), DOI 10.1038/s41586-019-1335-8 — doi.org/10.1038/s41586-019-1335-8
  4. Swain M. C., Cole J. M. "ChemDataExtractor: A Toolkit for Automated Extraction of Chemical Information from the Scientific Literature", J. Chem. Inf. Model. 56, 1894–1904 (2016), DOI 10.1021/acs.jcim.6b00207 — doi.org/10.1021/acs.jcim.6b00207
  5. Weston L. et al. "Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature", J. Chem. Inf. Model. 59, 3692–3702 (2019), DOI 10.1021/acs.jcim.9b00470 — doi.org/10.1021/acs.jcim.9b00470
  6. Huang S., Cole J. M. "A database of battery materials auto-generated using ChemDataExtractor", Sci. Data 7, 260 (2020), DOI 10.1038/s41597-020-00602-2 — doi.org/10.1038/s41597-020-00602-2
  7. Beltagy I., Lo K., Cohan A. "SciBERT: A Pretrained Language Model for Scientific Text", arXiv:1903.10676 (2019) — arxiv.org/abs/1903.10676
  8. Trewartha A. et al. "Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science", Patterns 3, 100488 (2022), DOI 10.1016/j.patter.2022.100488 — doi.org/10.1016/j.patter.2022.100488
  9. Gupta T., Zaki M., Krishnan N. M. A., Mausam "MatSciBERT: A materials domain language model for text mining and information extraction", npj Comput. Mater. 8, 102 (2022), DOI 10.1038/s41524-022-00784-w — doi.org/10.1038/s41524-022-00784-w
  10. Dagdelen J. et al. "Structured information extraction from scientific text with large language models", Nat. Commun. 15, 1418 (2024), DOI 10.1038/s41467-024-45563-x — doi.org/10.1038/s41467-024-45563-x
  11. Polak M. P., Morgan D. "Extracting accurate materials data from research papers with conversational language models and prompt engineering", Nat. Commun. 15, 1569 (2024), DOI 10.1038/s41467-024-45914-8 — doi.org/10.1038/s41467-024-45914-8
  12. Zheng Z. et al. "ChatGPT Chemistry Assistant for Text Mining and the Prediction of MOF Synthesis", J. Am. Chem. Soc. 145, 18048–18062 (2023), DOI 10.1021/jacs.3c05819 — doi.org/10.1021/jacs.3c05819
  13. Jablonka K. M. et al. "14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon", Digital Discovery 2, 1233–1250 (2023), DOI 10.1039/D3DD00113J — doi.org/10.1039/D3DD00113J
  14. Mishra V. et al. "Foundational Large Language Models for Materials Research", arXiv:2412.09560 (2024) (preprint; a journal version was not found) — arxiv.org/abs/2412.09560
  15. Xie T. et al. "DARWIN Series: Domain Specific Large Language Models for Natural Science", arXiv:2308.13565 (2023) (preprint) — arxiv.org/abs/2308.13565
  16. Niyongabo Rubungo A. et al. "LLM-Prop: predicting the properties of crystalline materials using large language models", npj Comput. Mater. 11, 186 (2025), DOI 10.1038/s41524-025-01536-2 (abstract checked in arXiv:2310.14029) — arxiv.org/abs/2310.14029
  17. Gruver N. et al. "Fine-Tuned Language Models Generate Stable Inorganic Materials as Text", arXiv:2402.04379 (2024) — arxiv.org/abs/2402.04379
  18. Song Y. et al. "MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling", arXiv:2305.08264 (2023) — arxiv.org/abs/2305.08264
  19. Zaki M., Jayadeva, Mausam, Krishnan N. M. A. "MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models", arXiv:2308.09115 (2023); published in Digital Discovery 3, 313–327 (2024), DOI 10.1039/D3DD00188A (figures in this article are from the arXiv abstract) — arxiv.org/abs/2308.09115
  20. Mirza A. et al. "A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists", Nat. Chem. 17, 1027–1034 (2025), DOI 10.1038/s41557-025-01815-x — doi.org/10.1038/s41557-025-01815-x
  21. Raccuglia P. et al. "Machine-learning-assisted materials discovery using failed experiments", Nature 533, 73–76 (2016), DOI 10.1038/nature17439 — doi.org/10.1038/nature17439
  22. Lowe D. "Chemical reactions from US patents (1976-Sep2016)", figshare dataset (2017), DOI 10.6084/m9.figshare.5104873.v1 (CC0) — doi.org/10.6084/m9.figshare.5104873.v1
  23. Wilkinson M. D. et al. "The FAIR Guiding Principles for scientific data management and stewardship", Sci. Data 3, 160018 (2016), DOI 10.1038/sdata.2016.18 — doi.org/10.1038/sdata.2016.18
  24. ISO/IEC "ISO/IEC 22989:2022 Information technology — Artificial intelligence — Artificial intelligence concepts and terminology" (ISO page) — iso.org/standard/74296.html
  25. ISO/IEC "ISO/IEC 23053:2022 Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)" (ISO page) — iso.org/standard/74438.html
  26. ISO/IEC "ISO/IEC 5259-1:2024 Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 1: Overview, terminology, and examples" (ISO page) — iso.org/standard/81088.html
  27. ISO/IEC "ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system" (ISO page) — iso.org/standard/42001
  28. NIST "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", NIST AI 100-1 (January 2023), DOI 10.6028/NIST.AI.100-1 — doi.org/10.6028/NIST.AI.100-1
  29. NIST "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", NIST AI 600-1 (26 July 2024), DOI 10.6028/NIST.AI.600-1 — doi.org/10.6028/NIST.AI.600-1

14. Claim-to-source audit

Claim in the textBasisClass
Solid-state dataset: 4,204,170 papers, 6,218,136 paragraphs in experimental sections, 188,198 inorganic-synthesis paragraphs, 53,538 solid-state paragraphs, 15,144 paragraphs giving a balanced reaction (28%), 19,488 reactions in the dataset. A check of 100 paragraphs (30 not sufficient to rebuild a reaction; 93% at chemistry level, 51% for all items, 64% without heating and mixing conditions). Permission from six publishers. The choice to favour precision over recallReference 1 https://doi.org/10.1038/s41597-019-0224-1Sourced
Text-mining review: PDF structure recovery below about 80% F1; TDM agreements needed; text mining in materials science at an early stage; bias toward successes and scarce publication of failures; extraction from tables and figures needs other technologyReference 2 https://doi.org/10.1016/j.isci.2021.102155Sourced
Structured databases hold a small fraction of literature knowledge; word embeddings captured the periodic-table structure without labels and recommended functional materials several years before discovery (as stated)Reference 3 https://doi.org/10.1038/s41586-019-1335-8Sourced
ChemDataExtractor toolkit; the battery database (292,313 records from 229,061 papers); named-entity extraction at 87% F1 on 3.27 million abstracts with over 80 million entitiesReference 4 https://doi.org/10.1021/acs.jcim.6b00207
Reference 6 https://doi.org/10.1038/s41597-020-00602-2
Reference 5 https://doi.org/10.1021/acs.jcim.9b00470		Sourced
SciBERT overview. MatBERT 1-12% better than other BERT-base models, BiLSTM above BERT. MatSciBERT reported above SciBERT on three tasksReference 7 https://arxiv.org/abs/1903.10676
Reference 8 https://doi.org/10.1016/j.patter.2022.100488
Reference 9 https://doi.org/10.1038/s41524-022-00784-w		Sourced
LLM-NERRE: fine-tuning GPT-3 and Llama-2, 100-500 examples, three tasks, F1 on MOF and general tasks about 30% below the dopant task, best F1 0.537 (exact match, a lower bound), a hallucination example (adding SiO₂ to "silica")Reference 10 https://doi.org/10.1038/s41467-024-45563-xSourced
ChatExtract: GPT-4 at 90.8% precision and 87.7% recall on bulk modulus (100 sentences, 179 data points); 634 values against a 721-entry reference for critical cooling rate at 76.9%/63.1%; 91.9%/84.2% for the standardized version (the abstract gives 91.6%/83.6%); 70% of sentences multi-valued; confusion of similar properties; the question designReference 11 https://doi.org/10.1038/s41467-024-45914-8Sourced
ChemPrompt: 26,257 synthesis parameters from about 800 MOFs, precision, recall and F1 of 90-99%, a crystallization prediction model above 86% accuracyReference 12 https://doi.org/10.1021/jacs.3c05819Sourced
Hackathon: prototypes for property prediction, interfaces, knowledge extraction and education built in under two days (as stated)Reference 13 https://doi.org/10.1039/D3DD00113JSourced
Machine learning using failed experiments from lab notebooks (Raccuglia et al.)Reference 21 https://doi.org/10.1038/nature17439Sourced
Dataset of chemical reactions text-mined from US patents (1976-September 2016, CC0); the note that duplicates are frequentReference 22 https://doi.org/10.6084/m9.figshare.5104873.v1Sourced
DARWIN (over 60,000 instruction examples), LLaMat (continued pretraining; the Llama-2 version better than the Llama-3 version as stated), LLM-Prop (property prediction from text, TextEdge data), crystal generation by fine-tuned LLaMA-2 (49% versus 28% predicted metastable)Reference 15 https://arxiv.org/abs/2308.13565
Reference 14 https://arxiv.org/abs/2412.09560
Reference 16 https://arxiv.org/abs/2310.14029
Reference 17 https://arxiv.org/abs/2402.04379		Sourced
MatSci-NLP (seven tasks). MaScQA (650 questions, GPT-4 about 62%, about 64% of errors conceptual and 36% computational). ChemBench (over 2,700 questions; best models above the best chemists on average yet wrong on basic tasks and overconfident)Reference 18 https://arxiv.org/abs/2305.08264
Reference 19 https://arxiv.org/abs/2308.09115
Reference 20 https://doi.org/10.1038/s41557-025-01815-x		Sourced
The FAIR Guiding Principles (2016)Reference 23 https://doi.org/10.1038/sdata.2016.18Sourced
Numbers, titles and status (published) and issue dates of ISO/IEC 22989:2022, 23053:2022, 5259-1:2024 and 42001:2023 (July 2022, June 2022, July 2024, December 2023)Reference 24 https://www.iso.org/standard/74296.html
Reference 25 https://www.iso.org/standard/74438.html
Reference 26 https://www.iso.org/standard/81088.html
Reference 27 https://www.iso.org/standard/42001		Sourced
NIST AI RMF 1.0 (January 2023, voluntary). NIST AI 600-1 (26 July 2024, definition of confabulation)Reference 28 https://doi.org/10.6028/NIST.AI.100-1
Reference 29 https://doi.org/10.6028/NIST.AI.600-1		Sourced
4,204,170 papers × 10 minutes = about 700,000 hours; / 2,000 hours = about 350 person-years. The step ratios (3.0%, 28.4%, 28.3%). At 90% precision about 100 of 1,000 are wrong; at 84% recall about 16% are missing. The precision/recall/F1 example (72/80, 72/100, 0.80)Our calculation (Python; 10 minutes and 2,000 hours are illustrative assumptions)Our calculation
Evaluation of extracting material properties or compositions from patents, repeat tests with the latest LLMs, model licences, current publisher TDM terms, an international standard specific to materials text extraction, peer-reviewed versions of LLaMat and DARWINNot confirmed in the primary sources reachable in this survey, so not statedCommentary
The readings "use as candidates", "decide the schema first", "record version, date and prompt", "include your own failure records" and the traffic-light groupingOur own arrangement and commentary based on published material; not a view stated by the researchers or institutionsCommentary
Figs. 2, 3, 5, 7 and 8 are conceptual diagrams; Figs. 4 and 6 plot reported values, with ratios from our calculation; Figs. 1 and 9 are slots for AI-generated images; the sentence in Fig. 3 was written for this article and is not quoted from a real paperNote by this articleCommentary

Last updated: 22 October 2026. Sources are limited to primary material (peer-reviewed papers, arXiv papers, ISO and NIST pages). Worked numbers are our own calculations, not results of an actual extraction. Repeat tests with the latest LLMs, evaluation of extracting materials data from patents, model licences and an international standard specific to materials text extraction could not be confirmed and are not stated. Figs. 1 and 9 are slots for AI-generated images; the other figures are explanatory diagrams or plots of reported values.

🌐 Japanese