1. What imitation learning and reinforcement learning are (the short version)

• Imitation learning is like learning by watching: a person’s demonstrations (what was seen, what action was taken) are treated as the right answer, and a policy is built by supervised learning. Having right answers makes learning fast, but it makes the policy weak in situations the demonstrations did not coverSourced[Ref. 1]
• Reinforcement learning is like trying things and being given a score: no right action is provided; the policy is improved by trial and error so that the score of the result (the reward) gets larger. It may go beyond what people can show, but it tends to need a huge number of trialsSourced[Ref. 9, 10]
• Doing everything on the real robot is heavy: so many trials are run in parallel in simulation and the result is moved to the real robot (sim2real). One report has a quadruped’s walking policy learned in about 20 minutes on one GPUSourced[Ref. 13]. But then the gap between simulation and the real machine (friction, for one) becomes the next wall

2. Three routes by which a robot acquires a skill

Suppose you want a robot to take a part out of a box and fit it into a hole. Instead of a person programming the motion (teaching coordinates one by one), there are broadly two ways to build a policy by learning.

The “strengths and weaknesses” rows are this article’s arrangement (commentary). Which method is better depends on the task, data and computing resources, and we do not rank them.

3. Imitation learning — learning “what is seen → what to do” from examples

(1) The simplest form: behavior cloning

At each demonstration, record what the robot could see (the observation) and the action the person took. Collect many such (observation, action) pairs and fit the mapping “in this observation, this action” with a neural network. That is behavior cloning. It has the same shape as ordinary supervised learning, with the person’s action as the “right answer”.

As an early well-known example, Bojarski et al. reported an end-to-end approach that learns from human driving data to map the pixels of a front camera to steering commands [Ref. 2]Sourced.

(2) Recent examples in robot manipulation

• ACT (ALOHA): from low-cost hardware and teleoperated demonstrations, it proposed generating actions several steps at a time (action chunking). It reports learning six difficult tasks, such as opening the lid of a translucent container and inserting a battery, at 80-90% success with only 10 minutes of demonstrations. The paper names errors that compound over time and human demonstrations that are not constant (non-stationary) as challenges [Ref. 3]Sourced
• Diffusion Policy: represents the policy as a diffusion process that extracts actions from noise. It reports an average improvement of 46.9% over existing methods on 12 tasks from four manipulation benchmarks (a comparison under the paper’s settings) [Ref. 4]Sourced
• robomimic: compared six offline learning methods on five simulated tasks and three real multi-stage tasks and analysed matters such as dependence on demonstration quality and variation with the time at which training is stopped [Ref. 5]Sourced

4. The pitfall of imitation learning — a small slip grows

Behavior cloning has the same form as supervised learning, so why is it hard? Because the robot’s action changes what it sees next. In ordinary image classification, a wrong judgement on one image does not change the next image. For a robot, a slightly wrong decision shifts its position a little, and what it sees next differs from the demonstrations.

Ross et al. describe this as a problem in which the assumption of independent and identically distributed data fails. The result is that a policy with error rate ε on the demonstrator’s situations can make as many as T²ε mistakes over a T-step task on the situations it induces itself. They also show examples in which this bound is reached in the worst case [Ref. 1]Sourced.

Checking with numbers

Assume (this is an assumption) a per-step error rate of ε = 1%. Here is how the number of mistakes changes with the length of the task (the number of steps) T (Our calculation).

Our calculation. ε = 1% is an assumed value. T²ε is the bound of Ross et al. [Ref. 1] and is a worst-case value (not the typical number of mistakes of a real policy). A policy cannot make more than T mistakes in T steps, so once T²ε exceeds T (that is, ε > 1/T) the bound tells us nothing.

Examples of countermeasures

• DAgger: run the learned policy, have an expert label the situations the policy itself passed through with the right actions, add them to the data and retrain. Repeating this brings the situations the policy actually enters into the demonstrations [Ref. 1]Sourced
• Output several steps of actions at once (ACT): this aims to reduce the number of per-step decisions and so limit compounding. The ACT paper names compounding errors as a challenge and addresses it with this approach of generating action sequences [Ref. 3]Sourced
• Bring in human corrections: there are methods where a person corrects the policy’s motion midway and the corrections are used for learning; HIL-SERL combines demonstrations and human corrections with reinforcement learning [Ref. 11]Sourced

5. Reinforcement learning — finding the way from the score of the result alone

In reinforcement learning, the system learns through repetition: the policy acts, and receives an observation and a reward from the environment (the robot, objects and the laws of physics). No right action is provided; the only clue is a score such as “+1 if it worked”. It is like playing a game over and over with only the score to go on, without being told the strategy (the limit of the analogy: a game does not break the machine when you fail).

(1) Reward and return

The score at each step is the reward, and the sum of rewards over time is the return. Later rewards are counted smaller by a discount factor γ (between 0 and 1). For example, if the reward for success comes only 4 steps later, a γ of 0.9 gives a return of 0.9⁴ = 0.656 and a γ of 0.5 gives 0.0625Our calculation. With a small discount factor the future counts for little, so tasks where “the reward comes only at the end of a long procedure” become hard to learn (commentary).

(2) Representative methods

• PPO is a family of policy-gradient methods that collects data by interacting with the environment and updates a “surrogate objective” over several epochs of minibatches. It is reported to be simple to implement and to strike a good balance with sample complexity [Ref. 6]Sourced
• SAC is an off-policy method that can reuse data; it reports improved stability compared with earlier methods that had suffered from high sample complexity and brittle convergence [Ref. 7]Sourced

(3) The design of the reward decides what is learned

A reward turns “what counts as good” into a number. If the reward has a loophole, the policy may earn points in a way that was not intended (reward hacking). Amodei et al. list this as one cause of accidents in machine-learning systems [Ref. 8]Sourced. For example, if the reward is tied to “how high the part is lifted”, a policy might be found that flings the part up without grasping it (this is a hypothetical example of ours, not a result from a paper).

6. How long does trial and error take on a real robot?

Let us check by orders of magnitude why reinforcement learning is called heavy. First an assumption: the real robot needs 1 million steps of trials and control runs 10 times a second. 1 million / 10 = 100,000 seconds, about 27.8 hours. For 10 million steps at 50 times a second, it is 200,000 seconds, about 55.6 hoursOur calculation. And that is the time if the robot never stops moving; the time to put objects back, repair broken parts and so on is not included (commentary).

There are also reports of reinforcement learning on real robots.

• To learn grasping from camera images alone, Levine et al. report collecting over 800,000 grasp attempts over two months using 6 to 14 robotic arms [Ref. 9]Sourced
• QT-Opt reports training on over 580,000 real grasp attempts and reaching a 96% grasp success rate on unseen objects [Ref. 10]Sourced
• HIL-SERL reports that by bringing in human demonstrations and corrections it brought tasks such as precision assembly to high success rates in 1 to 2.5 hours of training, and states an average 2x improvement in success rate over imitation-learning baselines [Ref. 11]Sourced

The three reports differ in task, robot, whether humans helped and how success is defined, so neither superiority nor training time can be compared. They are lined up only to give a sense of scale: “on a real robot, a few hours can be enough in some cases”, “but pure trial and error without human demonstrations or corrections can reach hundreds of thousands of attempts” (commentary).

7. Learn in simulation, move to the real machine (sim2real)

If real-machine trials are heavy, the natural idea is to practise inside the computer and then move to the real machine. The Isaac Gym paper reports that running physics simulation and policy training together on a GPU gave a speed-up of two to three orders of magnitude over conventional CPU-based simulators [Ref. 12]Sourced.

(1) Examples with legged robots

• Rudin et al. report that with massively parallel simulation on a single GPU, ANYmal learned a policy for walking on flat terrain in under four minutes and on uneven terrain in twenty minutes, and that they transferred it to the real robot to validate it [Ref. 13]Sourced
• Hwangbo et al. report moving a policy trained in simulation to ANYmal and achieving precise following of velocity commands, faster running than before and recovery from falls in complex configurations [Ref. 14]Sourced
• Lee et al. report that a policy trained by reinforcement learning only in simulation, using only the body’s own sensing (proprioception), walked robustly in natural environments such as mud, snow and rubble [Ref. 15]. Miki et al. report a policy that integrates exteroceptive perception of the terrain with proprioception [Ref. 16]Sourced

(2) An example with drones

Swift by Kaufmann et al. is an autonomous drone that combines deep reinforcement learning in simulation with data collected in the real world. In real head-to-head races against three human champions (including world champions of two international leagues), it is reported to have won several races against each of them and to have recorded the fastest race time [Ref. 17]Sourced. These are results of competition on a defined course, not a demonstration of performance on general flight tasks (commentary).

(3) Reducing the gap: domain randomisation

A representative countermeasure against the gap between simulation and reality (the reality gap) is domain randomisation. The simulator’s appearance and physical values are varied every time during training so that “reality looks like just one more variation”. Tobin et al. report that a real-world object detector accurate to 1.5 cm could be learned from simulator images with only non-realistic random textures [Ref. 18]. OpenAI reports that in-hand manipulation with a Shadow Dexterous Hand was learned only in simulation with randomised friction coefficients and object appearance, and moved to the real machine without human demonstrations [Ref. 19]. A follow-up study took on manipulating a Rubik’s cube with a robot hand using automatic domain randomisation (ADR) [Ref. 20]Sourced.

8. A materials engineer’s view: the “spread” of material properties and the “ageing” of real machines

9. Choosing and combining — which route to take

The “route that tends to fit” column is this article’s arrangement (commentary), not a recommendation by the papers. Real methods combine several routes.

10. Limits, and what could not be confirmed / summary

(1) Comparing results in papers

Success rates and training times are values reported for the robots, tasks and conditions each paper chose. We do not rank methods by lining up numbers from different conditions. We did not confirm independent third-party reproduction.

(2) Calculations from assumed values

The numbers in Chapters 4 and 6 are arithmetic on assumptions we made (ε = 1%, 1 million steps and so on), not the performance of any real policy or robot. The bound of Ross et al. is a worst-case value, not a typical number of mistakes.

(3) The text of standards

For ISO standards we confirmed only number, title and stage and did not read the text. We could not confirm a published standard that directly defines how to assess the safety of learned policies. ISO/IEC TS 22440, on functional safety of AI, is at the committee-draft stage.

(4) What we did not cover

How large pre-trained models relate to imitation and reinforcement learning is covered in our article on robot foundation models and VLA, how to collect demonstrations in our article on teleoperation and data collection, and the details of simulators in our article on robot simulation and digital twins. We do not cover humanoid products or market size.

11. How to read the yardsticks and standards

The basic yardstick for a learned policy is the success rate on a task. But a success rate changes with the number of trials, how objects are placed, the environment and who evaluates. The calculation of the width (confidence interval) of a success rate from few trials is in Chapter 6 of our article on world models and embodied AI. Here we list the benchmarks used to evaluate imitation and reinforcement learning, and the safety standards (checked 25 October 2026).

The arrangement of the table is Our calculation (organising). Numbers, titles, dates and stages of ISO standards were checked on the official ISO pages (25 October 2026). The text of the standards is paid, so we stay with titles and stages.

Notes on reading (for materials engineers)

• Demonstrations, benchmarks and the shop floor are different things: success rates in papers are values for chosen tasks in tidy environments. Re-measure with your own parts, floor, lighting and a worn machine
• Reported values and independent verification are different: training times and success rates are mostly measured by the research group that developed the method. Check whether there has been independent reproduction
• Is a standard a requirement, information, or in preparation? ISO 10218 sets safety requirements, ISO/IEC TR 5469 provides information, and ISO/IEC TS 22440 is in preparation. “Conforms to a standard” does not mean a learned policy is safe
• A success rate from few trials has a wide range: a “90%” from 9 of 10 trials is not the same as a “90%” from 900 of 1,000 (Chapter 6 of our article on world models and embodied AI)

12. Glossary

13. References

1. Ross S., Gordon G. J., Bagnell J. A. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning” (DAgger), arXiv:1011.0686 (2010; AISTATS 2011) — arxiv.org/abs/1011.0686
2. Bojarski M. et al. “End to End Learning for Self-Driving Cars”, arXiv:1604.07316 (2016) — arxiv.org/abs/1604.07316
3. Zhao T. Z. et al. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware” (ALOHA, ACT), arXiv:2304.13705 (2023) — arxiv.org/abs/2304.13705
4. Chi C. et al. “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion”, arXiv:2303.04137 (2023) — arxiv.org/abs/2303.04137
5. Mandlekar A. et al. “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation”, arXiv:2108.03298 (2021) — arxiv.org/abs/2108.03298
6. Schulman J. et al. “Proximal Policy Optimization Algorithms”, arXiv:1707.06347 (2017) — arxiv.org/abs/1707.06347
7. Haarnoja T. et al. “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, arXiv:1801.01290 (2018) — arxiv.org/abs/1801.01290
8. Amodei D. et al. “Concrete Problems in AI Safety”, arXiv:1606.06565 (2016) — arxiv.org/abs/1606.06565
9. Levine S. et al. “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, arXiv:1603.02199 (2016) — arxiv.org/abs/1603.02199
10. Kalashnikov D. et al. “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”, arXiv:1806.10293 (2018) — arxiv.org/abs/1806.10293
11. Luo J. et al. “Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning” (HIL-SERL), arXiv:2410.21845 (2024) — arxiv.org/abs/2410.21845
12. Makoviychuk V. et al. “Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning”, arXiv:2108.10470 (2021) — arxiv.org/abs/2108.10470
13. Rudin N. et al. “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, arXiv:2109.11978 (2021) — arxiv.org/abs/2109.11978
14. Hwangbo J. et al. “Learning agile and dynamic motor skills for legged robots”, Science Robotics 4, eaau5872 (2019), DOI 10.1126/scirobotics.aau5872 — doi.org/10.1126/scirobotics.aau5872
15. Lee J. et al. “Learning quadrupedal locomotion over challenging terrain”, Science Robotics 5, eabc5986 (2020), DOI 10.1126/scirobotics.abc5986 — doi.org/10.1126/scirobotics.abc5986
16. Miki T. et al. “Learning robust perceptive locomotion for quadrupedal robots in the wild”, Science Robotics 7, eabk2822 (2022), DOI 10.1126/scirobotics.abk2822 — doi.org/10.1126/scirobotics.abk2822
17. Kaufmann E. et al. “Champion-level drone racing using deep reinforcement learning”, Nature 620, 982–987 (2023), DOI 10.1038/s41586-023-06419-4 — doi.org/10.1038/s41586-023-06419-4
18. Tobin J. et al. “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”, arXiv:1703.06907 (2017) — arxiv.org/abs/1703.06907
19. OpenAI et al. “Learning Dexterous In-Hand Manipulation”, arXiv:1808.00177 (2018) — arxiv.org/abs/1808.00177
20. OpenAI et al. “Solving Rubik’s Cube with a Robot Hand”, arXiv:1910.07113 (2019) — arxiv.org/abs/1910.07113
21. Fu J. et al. “D4RL: Datasets for Deep Data-Driven Reinforcement Learning”, arXiv:2004.07219 (2020) — arxiv.org/abs/2004.07219
22. Yu T. et al. “Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning”, arXiv:1910.10897 (2019) — arxiv.org/abs/1910.10897
23. ISO ISO 10218-1:2025 “Robotics — Safety requirements — Part 1: Industrial robots” (Edition 3, February 2025). Checked 25 October 2026 — iso.org/standard/73933.html
24. ISO ISO 10218-2:2025 “Robotics — Safety requirements — Part 2: Industrial robot applications and robot cells” (Edition 2, February 2025). Checked 25 October 2026 — iso.org/standard/73934.html
25. ISO/IEC ISO/IEC TR 5469:2024 “Artificial intelligence — Functional safety and AI systems” (Edition 1, January 2024). Checked 25 October 2026 — iso.org/standard/81283.html
26. ISO/IEC ISO/IEC CD TS 22440-1 “Artificial intelligence — Functional safety and AI systems — Part 1: Requirements” (committee draft; stage 30.60, close of comment period). Checked 25 October 2026 — iso.org/standard/89535.html

14. Claim-to-source audit

Last updated: 25 October 2026 / Sources are limited to primary material (arXiv papers, papers in Science Robotics and Nature, and official ISO pages). We could not read the text of the ISO standards because it is paid. The numerical examples are calculations from values we assumed and are not the performance of any real robot or policy. We could not confirm, and so do not state, a standard that directly defines the safety assessment of learned policies, statistics on failures and breakages in real-robot reinforcement learning, or manufacturers’ stated training times for specific robots. Figures 1 and 8 are ImageGen-generated conceptual illustrations; the other figures are explanatory concept diagrams or figures containing our calculations.