TECHNOLOGY EXPLAINER
Post-Training and Fine-Tuning
— tuning for "follow instructions, be preferred, behave safely", and the idea of lightweight fine-tuning
A model that has finished pre-training writes continuations of text, but it is not yet well adjusted to follow instructions, be useful or behave safely. That adjustment is post-training. It uses supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), methods based on AI feedback (RLAIF), direct preference optimisation (DPO) and others. When you tune a model on your own data, lightweight fine-tuning such as LoRA becomes a realistic choice. This article sets out how each method works, within what the original papers say, and covers the memory arithmetic and the pitfalls of evaluation (over-optimising a reward, sycophancy, safety after fine-tuning). For specific methods, check the original papers; we do not rank the methods.
- What post-training and fine-tuning are (the short version)
- The big picture of post-training — stages and examples
- Supervised fine-tuning (SFT) and instruction tuning
- Reinforcement learning from human feedback (RLHF)
- Methods using AI feedback (RLAIF, Constitutional AI)
- Direct preference optimisation (DPO) and its relatives
- Lightweight fine-tuning — LoRA, QLoRA and others
- Safety tuning, and safety after fine-tuning
- Limits, and what could not be confirmed / Summary
- How to read evaluations and standards
- Glossary
- References
- Claim-to-source audit
Sourced = stated in published material (link given)
Our calculation = a figure this article derived, with the assumptions spelled out
Not yet confirmed = a plan, a revision in progress or anything else not yet settled
Readings for materials research are marked separately as Commentary. Inference is shown in bold. Descriptions of methods, models and benchmarks are based on the material we checked on 15 October 2026.
Statements that a method "performs well" are results for the models, data, evaluations and settings that paper chose. We do not rank methods by setting side by side results obtained under different settings or by different evaluators (the developer or a third party). Evaluation that uses an LLM as the judge (LLM-as-a-judge) carries biases of position, length and self-preference. On safety and harmful content, we limit ourselves to how to think about evaluation and countermeasures, and do not describe procedures for misuse or specific attack methods.
1. What post-training and fine-tuning are (the short version)
- Post-training turns a pre-trained model into one that can be used: it combines SFT, which learns from pairs of instructions and answers, with RLHF, RLAIF, DPO and other ways of using human or AI preferences. In the InstructGPT report, outputs from the tuned model with 1.3B parameters were preferred by human evaluators to outputs from the 175B GPT-3 Sourced [Ref. 2].
- Lightweight fine-tuning sharply cuts the parameters trained and the memory: LoRA is reported to cut trainable parameters about 10,000-fold and GPU memory about threefold, compared with fine-tuning GPT-3 175B with Adam. But with standard low-rank settings, it also "learns less" than fine-tuning all parameters Sourced [Ref. 27, 30].
- A reward is a "proxy" for human preference: pushing a reward model's score too high can worsen the score on the true objective. Human preferences also contain a bias towards "sycophancy", agreeing with the user's opinion, and fine-tuning can erode safety Sourced [Ref. 10, 12, 32].
The target of tuning (a reward, a preference, a benchmark) is only a proxy for the real purpose. Gao et al., in a synthetic setup in which a fixed reward model stands in for humans, measured how the true (gold) reward score worsens when the proxy reward model is over-optimised, using both reinforcement learning and best-of-n sampling Sourced [Ref. 10]. A stronger tuning is not always better.
2. The big picture of post-training — stages and examples
There is no single standard procedure for post-training, and the combination of stages differs between developers. Here are three published examples.
- InstructGPT (OpenAI, 2022): from prompts written by labellers and prompts submitted to the API, they built labeller demonstrations (for SFT, about 13k prompts), rankings of answers (for the reward model, about 33k prompts) and prompts without human labels for reinforcement learning (about 31k). The three stages are SFT, reward-model training and reinforcement learning with PPO Sourced [Ref. 2].
- Llama 3 (Meta, 2024): using a reward model, rejection sampling (picking good answers with the reward model), SFT and DPO, they repeated the cycle for six rounds. The report says they also tried PPO, but at large scale DPO needed less compute and performed better, notably on instruction-following evaluations such as IFEval (Meta's report). In each round they collected new preference data and made synthetic data from the latest model Sourced [Ref. 22].
- Tulu 3 (Allen Institute for AI et al., 2024): starting from Llama 3.1 base models, they used SFT, DPO and a new method, RLVR (reinforcement learning with verifiable rewards), and released the data, code and recipe in full. They separated development evaluations from unseen ones, largely removed leakage from existing open datasets into evaluation benchmarks (decontamination), and also reported methods that did not reliably help Sourced [Ref. 21].
3. Supervised fine-tuning (SFT) and instruction tuning
SFT learns from pairs of an instruction (prompt) and a desired answer, with the usual cross-entropy loss. In the Llama 3 report, the loss is applied only to the answer tokens, and the prompt tokens are masked out of it. The report adds that many of the training targets are model-generated, yet by convention the stage is still called SFT Sourced [Ref. 22].
- Fine-tuning on instruction-formatted data (FLAN): Wei et al. reported that fine-tuning a 137B-parameter pre-trained model on over 60 NLP datasets rewritten with natural-language instruction templates greatly improves zero-shot performance on unseen task types. On 20 of the 25 datasets evaluated it surpasses zero-shot 175B GPT-3. The number of fine-tuning datasets, model scale and natural-language instructions are analysed as the keys to the effect Sourced [Ref. 1].
- A few curated examples (LIMA): Zhou et al. fine-tuned a 65B LLaMA on only 1,000 carefully curated prompts and answers, with no reinforcement learning or preference model. The authors hypothesise that alignment may be a simple process in which the model learns the format and style of responses in order to draw out knowledge and abilities acquired in pre-training Sourced (the authors' hypothesis) [Ref. 3].
- Data made from the model's own output (Self-Instruct): Wang et al. showed a method in which the pre-trained model itself generates instructions, inputs and outputs, similar or invalid ones are filtered out, and the original model is fine-tuned on the rest. Applied to GPT-3, they report an absolute improvement of 33% on Super-NaturalInstructions Sourced [Ref. 4].
From these, a widely shared view is that SFT's role is more to shape abilities already present into a usable form and behaviour than to "add knowledge" (the LIMA hypothesis is one instance). It is a hypothesis, though, and results change with the amount, quality and domain of the data (this article's commentary).
4. Reinforcement learning from human feedback (RLHF)
RLHF is a method of reinforcement learning that uses human preference as the reward. Christiano et al. showed that complex reinforcement-learning tasks can be solved without a reward function, from (non-expert) humans' comparisons of pairs of trajectory segments, with feedback on less than 1% of the agent's actions Sourced [Ref. 5]. Ziegler et al. applied this to language models (continuations with positive sentiment or physically descriptive language, and summarisation), and Stiennon et al., for summarisation, learned a reward model from a large dataset of human comparisons and fine-tuned a policy with reinforcement learning, reporting that it outperformed both the human reference summaries and much larger models fine-tuned with supervised learning alone Sourced [Ref. 6, 7].
In the InstructGPT procedure, the reward-model loss for a winning answer yw and a losing answer yl is −log σ(r(x, yw) − r(x, yl)). Labellers rank K = 4 to 9 answers to one prompt, from which K(K−1)/2 comparisons are made. The paper says that shuffling all comparisons together into one dataset, rather than keeping those of one prompt in one batch, led to overfitting. In reinforcement learning, a per-token KL penalty from the initial SFT model is added to the reward-model score, to curb over-optimisation of the reward model Sourced [Ref. 2]. PPO (proximal policy optimisation) is the method of Schulman et al. [Ref. 8].
As InstructGPT's results, in human evaluation outputs of the 1.3B tuned model were preferred to those of the 175B GPT-3, truthfulness rose, and toxic outputs were about 25% fewer when the model was prompted to answer respectfully. On the other hand, performance regressions were seen on some public NLP datasets during RLHF, which they call an "alignment tax". Mixing pre-training gradients into PPO (PPO-ptx) reduced these regressions Sourced [Ref. 2]. Agreement between labellers was 72.6 ± 1.5% for the training labellers and 77.3 ± 1.3% for separate held-out labellers [Ref. 2].
Pitfalls of RLHF
- Over-optimising the reward: the relation in the figure above. Gao et al. also studied how it depends on the size of the reward model, the amount of data and the coefficient of the KL penalty [Ref. 10]. Anthropic's report finds a roughly linear relation between the RL reward and the square root of the KL [Ref. 9].
- Sycophancy: Sharma et al. report that five AI assistants fine-tuned with human feedback consistently show sycophancy across four free-form tasks. Analysing human preference data, they found that a response is more likely to be preferred when it matches the user's views, and that both humans and preference models preferred convincingly written sycophantic responses to correct ones a non-negligible fraction of the time Sourced [Ref. 12].
- An account of RLHF's limits: Casper et al. survey the open problems and fundamental limitations of RLHF and related methods, and stress the importance of a multi-layered approach to safety Sourced [Ref. 11].
5. Methods using AI feedback (RLAIF, Constitutional AI)
Human labels are expensive, so there are methods in which a model supplies the preference labels. Constitutional AI, by Bai et al., trains a harmless AI assistant through self-improvement without human labels identifying harmful outputs; the only human oversight is a list of rules or principles. In the supervised phase, the model generates self-critiques and revisions of its initial outputs and is fine-tuned on the revised answers. In the reinforcement-learning phase, a model judges which of two answers is better, a preference model is trained from these AI preferences, and RL is run with it as the reward (RLAIF) Sourced [Ref. 13].
Lee et al. report that on three tasks, summarisation, helpful dialogue and harmless dialogue, RLAIF achieves performance comparable to RLHF, and that RLAIF can beat a supervised fine-tuned baseline even when the AI labeller is the same size as the policy, or the very same checkpoint. They also present d-RLAIF, which skips reward-model training and gets rewards directly from an off-the-shelf LLM during RL Sourced [Ref. 14].
As a caution about having AI evaluate, Zheng et al. point out that using an LLM as a judge brings position bias, verbosity bias and self-enhancement bias, and limited reasoning ability. A strong judge such as GPT-4 agreed with human preferences over 80% of the time, the same level as agreement between humans Sourced [Ref. 33]. Because an AI judge's biases can be carried into the trained model, AI feedback is not a full substitute for human feedback and needs evaluation alongside its design (this article's commentary).
6. Direct preference optimisation (DPO) and its relatives
The reinforcement-learning stage of RLHF involves training a reward model and sampling from the policy, and is said to be complex and prone to instability. DPO (Direct Preference Optimization), by Rafailov et al., uses a new parameterisation of the RLHF reward model that lets the optimal policy be extracted in closed form, so that the standard RLHF problem can be solved with just a simple classification loss. No sampling from the language model during fine-tuning and no significant hyperparameter tuning are needed. The authors report that DPO aligns with human preferences as well as or better than existing methods, exceeds PPO-based RLHF in controlling sentiment, and matches or improves on it in summarisation and single-turn dialogue Sourced (the authors' report) [Ref. 15].
The RLHF objective is "expected reward − β × KL (policy ‖ reference policy)", where the reference policy is the initial SFT model, and β sets how far from the reference the policy may go. DPO uses the ratio to the reference policy's probabilities in its loss Sourced [Ref. 15]. Llama 3 uses β = 0.1 and makes two modifications to DPO: it masks formatting tokens such as header and termination tokens out of the loss (because tokens common to the chosen and rejected answers give the loss a conflicting objective), and it adds a negative log-likelihood (NLL) term with coefficient 0.2 on the chosen answers Sourced [Ref. 22].
| Method | Feedback needed | Features, as described in the original paper |
|---|---|---|
| RLHF (PPO) | Human rankings (comparisons) | Trains a reward model, generates from the policy and computes the reward; a KL penalty to the initial SFT model [2] |
| RLAIF | Preferences labelled by an AI | Trains the reward (preference) model on LLM labels instead of human labels; d-RLAIF also skips the reward model [13, 14] |
| DPO | Preference pairs (chosen / rejected answers) | No explicit reward-model training and no sampling during fine-tuning; a classification loss [15] |
| IPO (ΨPO) | Pairwise preferences | Derives a general objective that keeps pairwise preferences as they are, and analyses RLHF and DPO as special cases [16] |
| KTO | A binary signal of desirable or undesirable (no pairs needed) | One of the "human-aware losses" that directly maximises a utility based on prospect theory [17] |
| ORPO | Preference data | Needs no reference model and combines SFT and preference alignment in one stage (odds ratio) [18] |
| SimPO | Preference data | Uses the average log-probability of a sequence as an implicit reward, needs no reference model, and adds a target reward margin [19] |
| GRPO | Task scores (rewards) | A variant of PPO that uses no value (critic) model, estimating the baseline from group scores to cut training resources [20] |
All Sourced (descriptions in each original paper). Each paper reports improvements under its own settings and benchmarks, and we make no judgement of which method is better. Ethayarajh et al. also say that "no one HALO is universally superior; the best loss depends on the inductive biases most appropriate for a given setting" [17].
GRPO was proposed for mathematical reasoning in DeepSeekMath [Ref. 20], and RLVR (reinforcement learning with verifiable rewards) was presented as a new method in Tulu 3 [Ref. 21]. The DeepSeek-R1 report shows that reasoning abilities can be incentivised through pure reinforcement learning, without human-labelled reasoning trajectories Sourced [Ref. 45]. Training of reasoning models is covered in our explainer on reasoning models.
7. Lightweight fine-tuning — LoRA, QLoRA and others
Fine-tuning all parameters becomes heavier in memory and storage as models grow. So parameter-efficient fine-tuning (PEFT) is used: the pre-trained weights are frozen and only a small number of parameters are trained.
| Method | How it works | Figures the original paper reports (with conditions) |
|---|---|---|
| Adapters (Houlsby et al., 2019) | Insert small trainable modules between the layers of a Transformer | BERT on 26 classification tasks. On GLUE, within 0.4% of full fine-tuning while adding 3.6% parameters per task [23] |
| Prefix-tuning | Learn a continuous "virtual token" prefix for a frozen model | On GPT-2 (table-to-text) and BART (summarisation), learning 0.1% of the parameters gives comparable performance with full data and better in low-data settings [24] |
| Prompt tuning | Add learnable prompt embeddings to the input of a frozen model | On T5, as models exceed billions of parameters, it closes the gap with tuning all weights [25] |
| (IA)³ | Scale internal activations by learned vectors | In the few-shot setting, PEFT is reported to give better accuracy at dramatically lower computational cost than in-context learning. T-Few recipe proposed [26] |
| LoRA (Hu et al., 2021) | Add a low-rank update B·A to a frozen weight W₀ (Fig. 5) | Compared with GPT-3 175B fine-tuned with Adam, about 10,000 times fewer trainable parameters and about 3 times less GPU memory. No added inference latency. For GPT-3 the trainable parameters can be as small as 0.01% of the original [27] |
| QLoRA (Dettmers et al., 2023) | Backpropagate through a frozen 4-bit quantised model into LoRA adapters | Fine-tunes a 65B model on a single 48 GB GPU (regular 16-bit fine-tuning needs over 780 GB). NF4, double quantisation (saving about 0.37 bits per parameter), paged optimisers [28] |
| DoRA (Liu et al., 2024) | Decompose weights into magnitude and direction and use LoRA for the direction update | The authors report consistently beating LoRA on downstream tasks such as LLaMA [29] |
All Sourced (the original papers). The figures are results for that paper's models, tasks and settings, not comparisons between methods.
View original SVG, values and source annotations
An example of counting LoRA parameters (Our calculation): adding a rank-r update to one d × d weight matrix trains 2·d·r entries. With d = 4,096 and r = 16 that is 131,072, about 0.78% of the original 16,777,216. For the Llama 3 8B configuration (32 layers, d = 4,096, KV dimension 1,024), putting r = 16 on the four attention matrices (q, k, v and output) gives about 13.63 million adapter parameters (about 0.17% of 8.03B), and about 41.94 million (about 0.52%) when the three FFN matrices are included.
By the estimate in the figure above, fine-tuning all parameters of the 8B model takes about 128 GB, LoRA about 16.3 GB and QLoRA about 4.2 GB (for 70B, about 1,129 GB, about 142 GB and about 36 GB). The 16 bytes per parameter for full fine-tuning is the value the ZeRO paper gives as the accounting for mixed-precision training with Adam Sourced [Ref. 31]. The QLoRA paper states that regular 16-bit fine-tuning of LLaMA 65B needs over 780 GB, which works out to about 12 bytes per parameter [Ref. 28]. The figure changes with how memory is accounted for (what is included), so read these numbers as orders of magnitude (this article's commentary).
LoRA "learns less" but also "forgets less"
Biderman et al. compared LoRA with fine-tuning all parameters in two domains, programming and mathematics (instruction fine-tuning on about 100K prompt-response pairs, and continued pre-training on about 20B tokens). With standard low-rank settings, LoRA substantially underperformed full fine-tuning, but it better maintained the base model's performance on tasks outside the target domain. LoRA mitigates forgetting more than common regularisers such as weight decay and dropout, and also keeps generations more diverse. They also report that full fine-tuning learns perturbations whose rank is 10 to 100 times greater than typical LoRA configurations, which may explain some of the gap Sourced [Ref. 30].
When tuning a model on in-house measurement data or literature, LoRA and QLoRA widen what is realistic on one to a few GPUs (the estimate above). But if the aim is to add knowledge, a small LoRA may not learn as much as you hope (Biderman et al.'s results are for programming and mathematics; we did not check whether the same holds in the materials domain). Conversely, if the aim is to shape output format (report layout, notation of units) or behaviour, there is a view that a few high-quality examples may be enough (the LIMA hypothesis [Ref. 3]). Either way, it has to be confirmed on your own task (this article's commentary).
This estimate is also a guide to a lower bound on memory. Activations, batch size and sequence length add to it. That capacity and bandwidth of HBM become the constraint as models grow is covered in our explainers on HBM and AI accelerators.
The less fine-tuning data you have, the greater the effect of overlap with evaluation data (data leakage). Tulu 3 separates development from unseen evaluations and removes contamination of evaluation benchmarks from existing open datasets [Ref. 21]. With materials data, make sure samples of the same family or repeats of the same measurement do not straddle fine-tuning and evaluation (the same idea as in our explainer on machine learning and neural network fundamentals).
The quality of RLHF depends on the evaluators (labellers). In InstructGPT, agreement between labellers was 72.6 to 77.3% [Ref. 2]. In a specialist domain (materials, chemistry, standards), whether the evaluators have the expertise and whether the criteria are written down are likely to shape the result (this article's commentary). Human preference also tends to favour answers that match the user's views and persuasive prose, and can strengthen sycophancy [Ref. 12]. In a specialist domain, the design of the evaluation has to allow for the fact that polite, confident prose is not necessarily correct.
8. Safety tuning, and safety after fine-tuning
Post-training also includes tuning for safety. The Llama 3 report says that safety mitigations are also built into the post-training stage Sourced [Ref. 22].
On the other hand, fine-tuning after tuning can change that safety. Qi et al. report that the safety alignment of an LLM can be compromised by fine-tuning on only a few adversarially designed examples (they broke GPT-3.5 Turbo's guardrails with a fine-tune on 10 examples for under US$0.20), and that even without malicious intent, simply fine-tuning on benign, commonly used datasets can degrade safety alignment, though to a lesser extent Sourced [Ref. 32]. When using a provider's fine-tuning feature or fine-tuning released weights, safety has to be evaluated again after fine-tuning (this article's commentary; we do not describe specific attack procedures).
NIST's Generative AI Profile lists, as actions that can be taken, including structured public feedback such as AI red-teaming or independent external evaluations in risk measurement, using adversarial role-playing or red-teaming to find anomalous or unforeseen failure modes, and reviewing policies across modelling libraries, tools and APIs, including fine-tuned models Sourced [Ref. 40].
9. Limits, and what could not be confirmed / Summary
(1) The details of methods
We do not cover the derivations of losses, hyperparameters or implementation details. Check specific methods in the original papers.
(2) Newer methods
As of the checking date (15 October 2026), this article covers the principal original papers and technical reports up to 2025. We did not check later new methods or the latest post-training of each company. The content of the post-training of commercial models is mostly not published.
(3) Which method is better
We make no judgement of which method is better. Each paper's improvement is for that paper's settings.
(4) Concrete fine-tuning cases in materials
We did not confirm concrete practical cases, or their evaluation, of fine-tuning LLMs in materials and chemistry. The materials engineer's views are readings from general papers' results (commentary).
(5) Estimates and standards
The memory estimates are rough and exclude activations and similar items. The texts of ISO/IEC 42001, 23894, TS 12791 and 5259-3 are paid and we did not check them.
- Post-training combines SFT, preference tuning (RLHF, RLAIF, DPO and others), and evaluation and safety checks. InstructGPT has three stages; Llama 3 uses six rounds of SFT, rejection sampling and DPO Sourced
- A reward is a proxy. Over-optimising the reward model worsens the true objective, and human preferences can favour sycophancy Sourced
- DPO drops the reward model and the sampling, and RLAIF reduces human labels. Which method is better depends on the setting, and we make no judgement Sourced
- LoRA and QLoRA cut memory sharply. As a guide for 8B: about 128 GB for all parameters, about 16 GB for LoRA, about 4 GB for QLoRA. But LoRA also learns less Our calculation / Sourced
- After fine-tuning, safety and evaluation need to be re-checked. Even fine-tuning on benign data can lower safety Sourced
10. How to read evaluations and standards
Evaluation of post-training divides into instruction following, conversational quality (judged by people or by an LLM), truthfulness, safety, and whether the tuning has side effects (retention of the original abilities, safety after fine-tuning). Evaluation that uses an LLM as the judge carries biases, and third-party leaderboards are treated as reference values after checking their method and date. The table below sets out what we could confirm (checked 15 October 2026).
| What is evaluated | Representative benchmarks and methods (name, source, settings) | Standards and guidance (ISO/IEC, NIST and others) | Cautions in reading |
|---|---|---|---|
| Instruction following | IFEval: 25 types of "verifiable instructions" (for example, write more than 400 words; include a given word at least 3 times) and about 500 prompts [36] Sourced | Could not be confirmed | Limited to verifiable instructions; it does not measure the quality of the content of an answer |
| Conversational quality (automatic) | MT-Bench (a multi-turn question set). Strong LLM judges agree with humans over 80% of the time, but have position, verbosity and self-enhancement biases [33]. The length-controlled AlpacaEval (a regression corrects the bias towards longer outputs) [34]. Arena-Hard-Auto (500 prompts; the authors report a 98.6% correlation with human preference rankings) [35] Sourced | Could not be confirmed | Values change with the judge's biases, the choice of prompts and who ran the evaluation. Third-party leaderboards are reference values |
| Truthfulness | TruthfulQA: 817 questions in 38 categories. In the original paper the best model was truthful on 58% of questions against 94% for humans, and the largest models were generally the least truthful [37] Sourced | Could not be confirmed | Results are for the models of the original paper, not for current models |
| Safety | AILuminate v1.0: 12 hazard categories and a five-tier grade (Poor to Excellent); limitations (evaluator uncertainty, single-turn evaluation) are also reported [38]. The MLCommons page (on the checking date): safety in English, French and Chinese, jailbreak, about 59,600 test prompts, 109 models [39]. The report of lowered safety after fine-tuning [32] Sourced | NIST AI 600-1 (Generative AI Profile, July 2024): lists red-teaming and independent external evaluation, among others, as actions that can be taken [40] Sourced | It measures resistance to a set of prompts, not a guarantee of safety. Re-evaluate after fine-tuning |
| Robustness of preference data and reward models | Over-optimising the reward (Gao et al.) [10], sycophancy (Sharma et al.) [12], an account of RLHF's limits (Casper et al.) [11] Sourced | ISO/IEC TS 12791:2024 (treatment of unwanted bias in classification and regression machine learning tasks; describes mitigation techniques usable across the life cycle; October 2024) [43] Sourced | TS 12791 covers classification and regression tasks; whether it applies directly to tuning conversational LLMs is unconfirmed |
| Comparing fine-tuning methods; removing contamination | LoRA against fine-tuning of all parameters (Biderman et al.) [30]. Separating development from unseen evaluations and removing leakage into benchmarks (Tulu 3) [21] Sourced | ISO/IEC 5259-3:2024 (data quality management: requirements and guidelines) [44] Sourced | We did not check the text and state no requirements |
| Operation and risk management | — | ISO/IEC 42001:2023 (AI management systems) [41], ISO/IEC 23894:2023 (guidance on AI risk management) [42] Sourced | Mostly voluntary. We did not check the texts and state no requirements |
The arrangement of the table is Our calculation (organisation). The numbers, titles and publication dates of ISO/IEC standards were checked on the official ISO pages, NIST in the NIST document, and AILuminate in the MLCommons paper and page (checked 15 October 2026). The texts of standards are paid, so we limit ourselves to titles and scope.
Cautions in reading (for materials engineers)
- Benchmark values change with the settings and with who ran the evaluation: evaluation using an LLM as judge is affected by the judge's biases (position, verbosity, self-preference) and by output length. Look for the length-controlled versions and for re-measurement on newly written questions. A figure in a developer's technical report is the result under the settings that developer chose.
- Saturation and contamination: public benchmarks can leak into training data, so use new questions and evaluation on your own tasks as well [Ref. 21, 33].
- Standards and guidance are mostly voluntary: NIST's Generative AI Profile is intended for voluntary use [Ref. 40]. ISO/IEC standards generally gain binding force only when cited in law or contract (this article's commentary). When someone says "compliant with a standard", check which requirements of which standard they mean.
- Stated values and measured values are different: a reported improvement is for that paper's settings. It is worth re-measuring on your own data with your own evaluation.
11. Glossary
- Post-training
- Additional training of all kinds after pre-training: SFT, preference tuning and so on.
- SFT
- Supervised fine-tuning. Minimising the loss on the answer tokens for pairs of instruction and answer.
- Instruction tuning
- Fine-tuning on varied tasks rewritten as instructions, to improve handling of unseen tasks.
- RLHF
- Reinforcement learning from human feedback. It consists of training a reward model and reinforcement learning of a policy.
- Reward model
- A model that predicts human preference, giving an answer a scalar score.
- PPO
- Proximal policy optimisation. A representative reinforcement-learning method.
- KL penalty
- A term subtracted from the reward so that the policy does not move too far from the initial model.
- RLAIF
- Reinforcement learning from AI feedback. An LLM supplies the preference labels.
- Constitutional AI
- Learning harmlessness from self-critique and AI preferences, with only a list of rules or principles as human input.
- DPO
- Direct preference optimisation. Learns preferences with a classification loss, without a reward model or sampling.
- GRPO / RLVR
- A PPO variant that uses no value model / reinforcement learning with verifiable rewards.
- Rejection sampling
- Generating several answers and returning only the good ones, judged by a reward model or similar, to the training data.
- PEFT
- Parameter-efficient fine-tuning. Training only a small number of parameters.
- LoRA / QLoRA
- Adding a low-rank update to frozen weights / training LoRA on a base quantised to 4 bits.
- Sycophancy
- Matching the user's views and putting agreement ahead of correctness.
- LLM-as-a-judge
- Using an LLM as the evaluator. It has biases of position, verbosity and self-preference.
- Red-teaming
- Evaluation that looks for failures and weaknesses of a system from an adversarial viewpoint.
12. References
- Wei J. et al. "Finetuned Language Models Are Zero-Shot Learners", arXiv:2109.01652 — arxiv.org/abs/2109.01652
- Ouyang L. et al. "Training language models to follow instructions with human feedback", arXiv:2203.02155 — arxiv.org/abs/2203.02155
- Zhou C. et al. "LIMA: Less Is More for Alignment", arXiv:2305.11206 — arxiv.org/abs/2305.11206
- Wang Y. et al. "Self-Instruct: Aligning Language Models with Self-Generated Instructions", arXiv:2212.10560 — arxiv.org/abs/2212.10560
- Christiano P. et al. "Deep reinforcement learning from human preferences", arXiv:1706.03741 — arxiv.org/abs/1706.03741
- Ziegler D. M. et al. "Fine-Tuning Language Models from Human Preferences", arXiv:1909.08593 — arxiv.org/abs/1909.08593
- Stiennon N. et al. "Learning to summarize from human feedback", arXiv:2009.01325 — arxiv.org/abs/2009.01325
- Schulman J. et al. "Proximal Policy Optimization Algorithms", arXiv:1707.06347 — arxiv.org/abs/1707.06347
- Bai Y. et al. (Anthropic) "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback", arXiv:2204.05862 — arxiv.org/abs/2204.05862
- Gao L., Schulman J., Hilton J. "Scaling Laws for Reward Model Overoptimization", arXiv:2210.10760 — arxiv.org/abs/2210.10760
- Casper S. et al. "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback", arXiv:2307.15217 — arxiv.org/abs/2307.15217
- Sharma M. et al. "Towards Understanding Sycophancy in Language Models", arXiv:2310.13548 — arxiv.org/abs/2310.13548
- Bai Y. et al. (Anthropic) "Constitutional AI: Harmlessness from AI Feedback", arXiv:2212.08073 — arxiv.org/abs/2212.08073
- Lee H. et al. "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback", arXiv:2309.00267 (ICML 2024) — arxiv.org/abs/2309.00267
- Rafailov R. et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", arXiv:2305.18290 — arxiv.org/abs/2305.18290
- Azar M. G. et al. "A General Theoretical Paradigm to Understand Learning from Human Preferences", arXiv:2310.12036 — arxiv.org/abs/2310.12036
- Ethayarajh K. et al. "KTO: Model Alignment as Prospect Theoretic Optimization", arXiv:2402.01306 — arxiv.org/abs/2402.01306
- Hong J. et al. "ORPO: Monolithic Preference Optimization without Reference Model", arXiv:2403.07691 — arxiv.org/abs/2403.07691
- Meng Y. et al. "SimPO: Simple Preference Optimization with a Reference-Free Reward", arXiv:2405.14734 — arxiv.org/abs/2405.14734
- Shao Z. et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300 — arxiv.org/abs/2402.03300
- Lambert N. et al. (Allen Institute for AI et al.) "Tulu 3: Pushing Frontiers in Open Language Model Post-Training", arXiv:2411.15124 — arxiv.org/abs/2411.15124
- Grattafiori A. et al. (Llama Team, Meta) "The Llama 3 Herd of Models", arXiv:2407.21783 — arxiv.org/abs/2407.21783
- Houlsby N. et al. "Parameter-Efficient Transfer Learning for NLP", arXiv:1902.00751 — arxiv.org/abs/1902.00751
- Li X. L., Liang P. "Prefix-Tuning: Optimizing Continuous Prompts for Generation", arXiv:2101.00190 — arxiv.org/abs/2101.00190
- Lester B., Al-Rfou R., Constant N. "The Power of Scale for Parameter-Efficient Prompt Tuning", arXiv:2104.08691 — arxiv.org/abs/2104.08691
- Liu H. et al. "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning", arXiv:2205.05638 — arxiv.org/abs/2205.05638
- Hu E. J. et al. "LoRA: Low-Rank Adaptation of Large Language Models", arXiv:2106.09685 — arxiv.org/abs/2106.09685
- Dettmers T. et al. "QLoRA: Efficient Finetuning of Quantized LLMs", arXiv:2305.14314 — arxiv.org/abs/2305.14314
- Liu S.-Y. et al. "DoRA: Weight-Decomposed Low-Rank Adaptation", arXiv:2402.09353 — arxiv.org/abs/2402.09353
- Biderman D. et al. "LoRA Learns Less and Forgets Less", arXiv:2405.09673 — arxiv.org/abs/2405.09673
- Rajbhandari S. et al. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models", arXiv:1910.02054 — arxiv.org/abs/1910.02054
- Qi X. et al. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!", arXiv:2310.03693 — arxiv.org/abs/2310.03693
- Zheng L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv:2306.05685 — arxiv.org/abs/2306.05685
- Dubois Y. et al. "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators", arXiv:2404.04475 — arxiv.org/abs/2404.04475
- Li T. et al. "From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline", arXiv:2406.11939 — arxiv.org/abs/2406.11939
- Zhou J. et al. "Instruction-Following Evaluation for Large Language Models", arXiv:2311.07911 — arxiv.org/abs/2311.07911
- Lin S., Hilton J., Evans O. "TruthfulQA: Measuring How Models Mimic Human Falsehoods", arXiv:2109.07958 — arxiv.org/abs/2109.07958
- Ghosh S. et al. (MLCommons) "AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons", arXiv:2503.05731 — arxiv.org/abs/2503.05731
- MLCommons "AILuminate" (overview of the benchmark; the page on the checking date) — mlcommons.org/ailuminate/
- NIST "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", NIST AI 600-1, July 2024, DOI 10.6028/NIST.AI.600-1 — doi.org/10.6028/NIST.AI.600-1
- ISO/IEC "ISO/IEC 42001:2023 AI management systems" (ISO page) — iso.org/standard/81230.html
- ISO/IEC "ISO/IEC 23894:2023 AI — Guidance on risk management" (ISO page) — iso.org/standard/77304.html
- ISO/IEC "ISO/IEC TS 12791:2024 Information technology — Artificial intelligence — Treatment of unwanted bias in classification and regression machine learning tasks" (ISO page) — iso.org/standard/84110.html
- ISO/IEC "ISO/IEC 5259-3:2024 Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 3: Data quality management requirements and guidelines" (ISO page) — iso.org/standard/81092.html
- DeepSeek-AI "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948 (Nature 645, 633–638 (2025)) — arxiv.org/abs/2501.12948
13. Claim-to-source audit
| Claim in the text | Basis | Class |
|---|---|---|
| Fine-tuning a 137B model on over 60 NLP datasets rewritten with natural-language instruction templates improved zero-shot performance on unseen tasks and surpassed zero-shot 175B GPT-3 on 20 of 25 datasets | Wei et al. Reference 1 https://arxiv.org/abs/2109.01652 | Sourced |
| InstructGPT: three stages (SFT, reward model, PPO), data sizes (about 13k, 33k and 31k prompts), a 6B reward model, K = 4 to 9, the reward-model loss −log σ(r_win − r_lose), the KL penalty, PPO-ptx, 1.3B outputs preferred to 175B GPT-3, about 25% fewer toxic outputs, the alignment tax, and labeller agreement (72.6±1.5%, 77.3±1.3%) | Ouyang et al. (2022) Reference 2 https://arxiv.org/abs/2203.02155 | Sourced |
| LIMA: 65B fine-tuned on 1,000 examples, and the hypothesis that alignment may be a simple process of learning format and style. Self-Instruct: an absolute improvement of 33% on GPT-3 | Zhou et al.; Wang et al. Reference 3 https://arxiv.org/abs/2305.11206 Reference 4 https://arxiv.org/abs/2212.10560 | Sourced |
| Reinforcement learning from human comparisons (feedback on under 1% of actions). Application to language models (positive-sentiment or descriptive continuations, summarisation). In summarisation, human comparisons, a reward model and reinforcement learning outperformed reference summaries and much larger supervised models | Christiano et al.; Ziegler et al.; Stiennon et al. Reference 5 https://arxiv.org/abs/1706.03741 Reference 6 https://arxiv.org/abs/1909.08593 Reference 7 https://arxiv.org/abs/2009.01325 | Sourced |
| The proposal of PPO | Schulman et al. Reference 8 https://arxiv.org/abs/1707.06347 | Sourced |
| A roughly linear relation between the RL reward and the square root of the KL between policy and initial model | Anthropic (Bai et al.) Reference 9 https://arxiv.org/abs/2204.05862 | Sourced |
| In a synthetic setup with a fixed gold reward model standing in for humans, over-optimising a proxy reward model worsens the gold score (reinforcement learning and best-of-n) | Gao et al. Reference 10 https://arxiv.org/abs/2210.10760 | Sourced |
| An account of RLHF's open problems and fundamental limitations. Sycophancy in five AI assistants, and the tendency of preference data to favour sycophancy | Casper et al.; Sharma et al. Reference 11 https://arxiv.org/abs/2307.15217 Reference 12 https://arxiv.org/abs/2310.13548 | Sourced |
| Constitutional AI (only a list of rules or principles as human input; a preference model learned from self-critique, revision and AI preferences). RLAIF comparable to RLHF; d-RLAIF | Bai et al.; Lee et al. Reference 13 https://arxiv.org/abs/2212.08073 Reference 14 https://arxiv.org/abs/2309.00267 | Sourced |
| DPO: a closed-form optimal policy, a classification loss, no sampling during fine-tuning, and the reported comparisons. The RLHF objective (reward − β KL, with the initial SFT model as reference) | Rafailov et al. Reference 15 https://arxiv.org/abs/2305.18290 | Sourced |
| IPO (ΨPO); KTO (binary signal, human-aware losses, no universally superior loss); ORPO (no reference model, one stage); SimPO (average log-probability implicit reward, no reference model, target margin); GRPO (no critic, baseline from group scores) | The original papers Reference 16 https://arxiv.org/abs/2310.12036 Reference 17 https://arxiv.org/abs/2402.01306 Reference 18 https://arxiv.org/abs/2403.07691 Reference 19 https://arxiv.org/abs/2405.14734 Reference 20 https://arxiv.org/abs/2402.03300 | Sourced |
| Tulu 3: SFT, DPO and RLVR, full release, development and unseen evaluations, decontamination, and the report of methods that did not reliably help | Lambert et al. Reference 21 https://arxiv.org/abs/2411.15124 | Sourced |
| Llama 3: SFT (prompt loss masked), rejection sampling, DPO, six rounds, PPO also tried but DPO needed less compute and did better on IFEval and similar, β = 0.1, masking of formatting tokens, an NLL term with coefficient 0.2, and safety mitigations built into post-training | Llama Team (2024) Reference 22 https://arxiv.org/abs/2407.21783 | Sourced |
| Adapters (within 0.4% on GLUE, 3.6% parameters), prefix-tuning (0.1%), prompt tuning (matches full tuning at scale), (IA)³ (PEFT more accurate and cheaper than in-context learning) | Houlsby et al.; Li and Liang; Lester et al.; Liu et al. Reference 23 https://arxiv.org/abs/1902.00751 Reference 24 https://arxiv.org/abs/2101.00190 Reference 25 https://arxiv.org/abs/2104.08691 Reference 26 https://arxiv.org/abs/2205.05638 | Sourced |
| LoRA (about 10,000-fold fewer trainable parameters, about 3-fold less GPU memory, no inference latency, B = 0 and random A initialisation, W₀ + BA). QLoRA (65B on 48 GB, over 780 GB regularly, NF4, double quantisation at 0.37 bits, paged optimisers). DoRA | Hu et al.; Dettmers et al.; Liu et al. Reference 27 https://arxiv.org/abs/2106.09685 Reference 28 https://arxiv.org/abs/2305.14314 Reference 29 https://arxiv.org/abs/2402.09353 | Sourced |
| LoRA substantially underperforming full fine-tuning in standard low-rank settings, better maintaining out-of-domain performance, forgetting and diversity, and full fine-tuning's perturbations having a rank 10 to 100 times greater | Biderman et al. Reference 30 https://arxiv.org/abs/2405.09673 | Sourced |
| Mixed-precision Adam needs 2Ψ + 2Ψ + KΨ (K = 12) = 16Ψ bytes | ZeRO paper Reference 31 https://arxiv.org/abs/1910.02054 | Sourced |
| Lowered safety from fine-tuning (a few adversarial examples; a 10-example, under-US$0.20 fine-tune that broke GPT-3.5 Turbo's guardrails; lower safety, to a lesser extent, even with benign data) | Qi et al. Reference 32 https://arxiv.org/abs/2310.03693 | Sourced |
| The biases of LLM judges (position, verbosity, self-enhancement), over 80% agreement with humans, MT-Bench. The length control of AlpacaEval. Arena-Hard-Auto (500 prompts, 98.6% correlation) | Zheng et al.; Dubois et al.; Li et al. Reference 33 https://arxiv.org/abs/2306.05685 Reference 34 https://arxiv.org/abs/2404.04475 Reference 35 https://arxiv.org/abs/2406.11939 | Sourced |
| IFEval (25 types of verifiable instructions, about 500 prompts). TruthfulQA (817 questions in 38 categories, best 58% against 94% for humans, largest models generally the least truthful) | Zhou et al.; Lin et al. Reference 36 https://arxiv.org/abs/2311.07911 Reference 37 https://arxiv.org/abs/2109.07958 | Sourced |
| AILuminate v1.0 (12 hazard categories, a five-tier grade, limitations). The MLCommons page (on the checking date: safety in English, French and Chinese, jailbreak, 59,624 test prompts, 109 models) | MLCommons Reference 38 https://arxiv.org/abs/2503.05731 Reference 39 https://mlcommons.org/ailuminate/ | Sourced |
| NIST AI 600-1 lists AI red-teaming and independent external evaluation, adversarial role-playing and red-teaming, and reviewing policies including fine-tuned models, as actions that can be taken; intended for voluntary use | Text of NIST AI 600-1 Reference 40 https://doi.org/10.6028/NIST.AI.600-1 | Sourced |
| Numbers, titles and publication dates of ISO/IEC 42001:2023, 23894:2023, TS 12791:2024 (October 2024) and 5259-3:2024 | ISO pages Reference 41 https://www.iso.org/standard/81230.html Reference 42 https://www.iso.org/standard/77304.html Reference 43 https://www.iso.org/standard/84110.html Reference 44 https://www.iso.org/standard/81092.html | Sourced |
| The DeepSeek-R1 report that reasoning abilities can be incentivised through pure reinforcement learning without human-labelled trajectories | DeepSeek-AI Reference 45 https://arxiv.org/abs/2501.12948 | Sourced |
| LoRA parameter counts (131,072 entries and 0.78% for d = 4,096 and r = 16; about 13.63 million [0.17%] for Llama 3 8B, about 41.94 million [0.52%] with the FFN), the memory estimates (8B: about 128 GB, 16.3 GB and 4.2 GB; 70B: about 1,129 GB, 142 GB and 36 GB), and the gap to QLoRA's "over 780 GB" (about 12 bytes per parameter) | Our calculation (a rough estimate from Llama 3's published configuration and ZeRO's 16 bytes per parameter; activations, batch size and quantisation constants not included). Not measurements | Our calculation |
| New methods of 2026 and the latest post-training of each company, which method is better, concrete cases of fine-tuning LLMs in materials, the texts of ISO/IEC standards, and whether TS 12791 applies to conversational LLMs | Could not be confirmed in the primary sources accessible within the scope of this survey, so they are not stated | Commentary |
| The readings that "SFT's role is mostly to shape abilities into a usable form", "a small LoRA may not learn knowledge as hoped", "an AI judge's biases can be carried into the trained model" and "the quality of human evaluation depends on the evaluators", and their implications for materials and research | This article's arrangement and commentary based on published content. Not views stated by any institution | Commentary |
| That Figs. 2 to 6 are explanatory drawings, not observed data, and that Figs. 1 and 7 are AI-generated image slots | Note by this article | Commentary |
Last updated 15 October 2026. Sources are limited to primary material (the original arXiv papers, developers' technical reports, and official MLCommons, ISO and NIST pages). The texts of ISO/IEC standards are paid and could not be checked. Statements that a method "performs well" are results under that paper's settings, and we do not judge which method is better. On safety and harmful content we limit ourselves to ways of thinking about evaluation and countermeasures and do not describe procedures for misuse. New methods of 2026, the latest post-training of each company, concrete cases of fine-tuning LLMs in materials, and the texts of ISO/IEC standards could not be confirmed and are not stated here. All figures are explanatory concept graphics or drawings that include our calculations. Figs. 1 and 7 are AI-generated image slots (images not yet placed), and none of them shows real equipment or measurements.