1. What robot vision is (the short version)

• “Seeing” is a chain of stages, each answering a different question: capture an image and distances with a camera and depth sensor → turn them into a set of 3D points (a point cloud) → find the objects → work out position and orientation → choose how to grasp. The stages answer “What is in the picture?”, “Where, and facing which way?” and “How should it be held?”
• Range sensors are not magic: the error of a two-camera method (stereo) grows with the square of the distanceSourced[Ref. 1]. Transparent objects are known to give standard 3D sensors values that look like the surface behind themSourced[Ref. 18]
• “Detected” is not “grasped”: the success rates of the stages multiply (98%, 95% and 90% give about 84%)Our calculation. Read the metrics that score box overlap (IoU, AP) separately from the success rate of actually grasping on a real robot

2. From “seeing” to “grasping” — the whole flow

Think of picking up a cup from a desk. A person, the instant they see it, knows “it is a cup”, “it is within reach”, “the handle faces this way” and “I should hold it here”, all without thinking. A robot splits this into stages and computes each one.

The method names are representative examples confirmed in papers; they do not indicate ranking or popularity. “Where errors tend to arise” is this article’s own arrangement (commentary).

Splitting into stages makes it easier to tell where a failure comes from. The price is that each later stage trusts the output of the one before, so an early error is passed straight on. The second half of this article puts numbers on that “carry-over”.

3. The parts of the eye — cameras and depth sensors

(1) An ordinary camera gives colour, but not distance directly

The image from an ordinary colour (RGB) camera carries colour and texture. Two objects can look the same size whether one is small and near or the other large and far, and a single image cannot settle which. So robots combine a camera with a distance-measuring sensor. A sensor that provides colour and distance together is called an RGB-D sensor (D stands for depth).

(2) Four ways to get distance

Here are the main approaches, with how they work and where they are strong or weak.

The explanations of how each works are this article’s general summary (commentary). The R200 configuration and the training of Depth Anything V2 come from the abstracts of the respective papers. We do not rank the methods.

(3) Checking how stereo works with a figure

Hold up a finger in front of your face and look at it with the right eye only, then the left eye only. The finger appears to jump sideways against the background, and the nearer the finger, the bigger the jump. Stereo vision counts that shift and turns it into distance.

Original SVG, detailed explanation and sources

Original SVG reference: Conceptual diagram (vector drawing). Two cameras looking at the same point see it shifted between the two images; the nearer the point, the larger the shift (disparity). The relation Z = f × B / d has the same form as in Keselman et al. [Ref. 1]. The lengths are for illustration, not the dimensions of any real camera.

As a formula, the distance Z is “focal length f × camera spacing B ÷ disparity d” (Z = f × B / d). The larger the disparity (the shift), the nearer the point [Ref. 1]. Here the focal length f and the disparity d are counted in pixels, while the spacing B and the distance Z are in metres.

4. Distance error grows with the square of distance — our calculation

Let us put numbers on how the error of a stereo method grows. First the intuition: in a distant landscape the left-eye and right-eye views barely differ. When the shift is small, a slight misreading makes a large difference in distance.

Keselman et al. differentiate the distance formula and show that the depth error grows as Z² / (f × B) times the disparity error [Ref. 1]. The disparity error is set by image quality and the matching method and can be treated as roughly constant, they write (about 0.1 pixel from their measurements). In other words, doubling the distance quadruples the error, and doubling the camera spacing B halves it.

The table below is calculated with assumed values: focal length f = 600 pixels and a disparity error of 0.1 pixel (Our calculation).

Our calculation. Assumed f = 600 pixels and disparity error = 0.1 pixel; error = Z² / (f × B) × 0.1. These are not the specifications of a real camera. The length covered by one pixel is the distance divided by the focal length (a pixel spans a width that grows in proportion to distance).

The same paper reports, as measurements on one unit (R200, 480 x 360 pixels, 30 Hz), 2.1 mm at 1 m and 8.0 mm at 2 m [Ref. 1]. This is the same order of magnitude as the assumed curve (2.4 mm at 1 m, 9.5 mm at 2 m), and the value at 2 m being about four times the value at 1 m agrees too. It is one example, though, and not a value for other devicesSourced.

What this calculation tells us

• The nearer you look, the better: the common arrangement of a camera on the wrist, moving close to the object, fits how the error grows (commentary)
• Small things seen from afar are “less than one pixel”: as the table shows, one pixel covers about 5 mm at 3 m, so a 3 mm screw is smaller than one pixelOur calculation
• A wider spacing reduces error, but at a cost: the camera becomes bigger, and nearby objects are harder to get into both views (commentary)

5. Point clouds — turning a depth image into points in space

A depth image has a distance in each pixel. We turn it into a set of 3D points (a point cloud), which is easier for a robot to handle. The calculation is simple and uses the lens model of a pinhole camera.

If the pixel position (u, v) and the distance Z are known, the 3D position (X, Y, Z) seen from the camera is X = (u − cx) × Z / f and Y = (v − cy) × Z / f. Here (cx, cy) is the pixel at the image centre and f is the focal length in pixels. For example, for a camera with f = 600 pixels and centre (320, 240), if the pixel (420, 300) has distance 0.8 m, then X = (420 − 320) × 0.8 / 600 = about 0.133 m and Y = (300 − 240) × 0.8 / 600 = 0.08 mOur calculation.

How heavy the data is

A point cloud made from a 640 x 480 pixel depth image has 307,200 points. Storing each point as three single-precision numbers (4 bytes x 3) takes about 3.7 MB, and at 30 frames per second about 111 MB/sOur calculation. Point clouds are convenient but heavy to process as they are, so preprocessing such as thinning out points or cutting out only the region of the object is used (commentary).

Aligning point clouds: ICP

A classic way to fix an object’s position and orientation by overlaying a 3D model of the object (a CAD model, for example) on the point cloud from the camera is ICP (iterative closest point). It is a method for registering 3D shapes reported by Besl and McKay in 1992, built on the idea of matching nearby points and nudging the shapes together step by step. What we confirmed in the paper is its title and bibliographic data; the description of the mechanism is this article’s general commentary [Ref. 3]Sourced. If the starting position is far off, the shapes may fail to line up (commentary).

6. What is in the picture — object detection and segmentation

(1) The difference between a box and a mask

Object detection marks objects in an image with rectangular boxes and labels. Segmentation goes finer and paints, for each pixel, which object it belongs to. Segmentation that separates objects of the same kind one by one is called instance segmentation. For grasping, a mask is more convenient than a box, because it lets you cut out a point cloud that follows the object’s outline (commentary).

(2) Representative methods (as described in the papers)

• YOLO treats detection as a regression problem in which one network outputs box positions and class probabilities in a single pass; the paper reports that the base model runs at 45 frames per second [Ref. 4]Sourced
• Mask R-CNN adds a branch that outputs masks to the box-prediction machinery, and is reported to run at about 5 frames per second [Ref. 5]Sourced
• DETR treats detection as a problem of directly outputting a set of objects, and the authors say it removes hand-designed components such as the step that removes duplicates [Ref. 6]Sourced
• Segment Anything (SAM) is a model that cuts out regions from prompts such as points or boxes, reported as trained on a dataset of over a billion masks on 11 million images [Ref. 7]. SAM 2 covers both images and video [Ref. 8]Sourced
• Grounding DINO is an “open-vocabulary” detector that looks for objects specified in words (category names or descriptive phrases) [Ref. 9]Sourced

The processing speeds in each paper are for the computer and image size that paper used, not a comparison under the same conditions. You cannot conclude from them which is faster or better (commentary).

(3) What counts as a “hit”: IoU and AP

Whether a detection is right is decided by how much the predicted box overlaps the true box. That overlap is the IoU (overlap area divided by combined area). The figure below shows a case where the box is slightly off and the IoU works out to 0.59.

According to the COCO evaluation page, the main metric AP is averaged over 10 IoU thresholds from 0.50 to 0.95 in steps of 0.05, and also over the 80 categories. It also reports AP at IoU 0.50 alone (the PASCAL VOC metric) and at the stricter 0.75, and AP split by object size into small (area under 32² pixels), medium and large (over 96²), computed with at most 100 detections per image [Ref. 11]Sourced. The COCO dataset contains 2.5 million labelled instances in about 328,000 images [Ref. 10]Sourced.

7. Where, and facing which way — 6D pose

An object’s position can be given by three numbers (x, y, z) and its orientation by three rotations (tilting forward and back, tilting left and right, and turning about its axis). Together these are six degrees of freedom, hence “6D pose”. Degrees of freedom (DOF) usually means “the number of directions in which something can move”, as in the rest of the series; here it means the number of independent values for an object’s position and orientation.

• PoseCNN finds an object’s centre in the image, predicts its distance from the camera to get the 3D position, and regresses the rotation as a quaternion. It also introduced a loss that handles symmetric objects, and released the YCB-Video dataset (21 objects, 92 videos, 133,827 frames) [Ref. 12]Sourced
• BOP is a benchmark for comparing methods that estimate the 6D pose of a rigid object from a single RGB-D image. It provides eight datasets and a common evaluation method, and evaluated 15 methods at the time of the paper [Ref. 14]Sourced
• FoundationPose is a foundation model reported to estimate and track the 6D pose of new objects without fine-tuning, given a CAD model or a small number of reference images [Ref. 15]Sourced

8. How to grasp — perception for grasping

Even when an object’s position and orientation are known, where and how to pinch it is a separate question. It depends on the hand’s shape (two fingers or suction), the object’s weight and how slippery its surface is. This is why research moved to predicting the grasp directly.

• Dex-Net 2.0 trained a network to predict the success probability of a grasp using 6.7 million synthetic point clouds and grasps made from 3D models. In over 1,000 real trials on an ABB YuMi, it reports a 93% success rate on eight known objects with difficult geometry, and 0.8 s to plan a grasp [Ref. 16]Sourced
• Contact-GraspNet generates grasps for a parallel-jaw hand (two plates that pinch) directly from a depth recording. Trained on 17 million simulated grasps, it reports a success rate above 90% on unseen objects in structured clutter, halving the failure rate compared with a recent state-of-the-art method [Ref. 17]Sourced
• ClearGrasp estimates the 3D shape of transparent objects from a single RGB-D image. Because transparent objects give standard 3D sensors noisy values resembling the surface behind them, it infers surface normals and masks and corrects the depth. The authors built a synthetic dataset of over 50,000 images and a real test set of 286 images [Ref. 18]Sourced

All these success rates are for the objects, placements, hands and number of trials each paper chose. “Eight known objects” for Dex-Net and “unseen objects in structured clutter” for Contact-GraspNet are different conditions, so the numbers cannot be lined up to say which is better (commentary). How to read success rates from few trials, with confidence intervals, is covered in Chapter 6 of our article on world models and embodied AI.

Stage success rates multiply

Assume, for illustration, “detection 98%, pose 95%, grasp 90%”. If the stages are independent, the overall success rate is 0.98 × 0.95 × 0.90 = about 84%. Which stage should be improved to raise the total the most?

• Raising detection from 98% to 99.5% brings the total to about 85.1% (+1.3 points)
• Raising the grasp from 90% to 95% brings the total to about 88.4% (+4.6 points)

Our calculation. The stage success rates are illustrative assumptions, with the stages assumed independent. They are not the results of any real system.

The conclusion is simple. Improving the weakest stage pays off most, and chasing the detection metric (AP) alone does not tell you the final success rate. In practice an error in one stage changes the failure rate of the next, so independence is an idealisation (commentary).

9. A materials engineer’s view 2: surface, lighting and dirt decide whether perception works

All the methods so far assume that “light hits the object and comes back”. So the object’s surface properties and the lighting strongly influence whether perception succeeds. This is the chapter where materials knowledge helps most.

Original SVG, detailed explanation and sources

Original SVG reference: Conceptual diagram (vector drawing). The same shape can be easy or hard to measure depending on how its surface returns light. That depth on transparent objects comes out as a noisy value resembling the surface behind is stated in the ClearGrasp paper [Ref. 18]; an example of a system using projected infrared light is in Keselman et al. [Ref. 1].

How each kind of surface looks

• Matte surfaces: light scatters in all directions, so the brightness looks similar from any angle and distance tends to be stable (commentary)
• Glossy surfaces (polished metal, glass-like finishes): light returns strongly in one direction, so the surface glares at one angle and looks dark at another. Holes or jumps may appear in the depth image (commentary)
• Transparent surfaces (glass, clear plastic): light passes through, so standard 3D sensors give noisy values resembling the surface behind [Ref. 18]Sourced
• Very dark surfaces (black rubber, carbon-containing materials): they absorb light and return little, which may make distance hard to obtain (this is this article’s general expectation, and we did not confirm it in primary sources)

10. Types of perception failure, and how to check for them

Here are the failures that tend to occur on site, sorted by cause. Use it as a checklist when trying this in your own process (this is this article’s arrangement, not statistics).

The table is this article’s arrangement (commentary). Actual failure rates vary greatly with the object, environment and equipment.

11. Limits, and what could not be confirmed / summary

(1) Accuracy of products and devices

We do not cover the accuracy or processing speed of specific cameras, depth sensors or robots as products. The distance-error table is the formula of Keselman et al. [Ref. 1] filled in with values we assumed; it is not the specification of a real device.

(2) Comparing results in papers

YOLO’s 45 frames per second, Dex-Net’s 93% success and Contact-GraspNet’s 90%-plus are values reported under the conditions each paper chose. We do not rank methods by lining up numbers from different conditions. We did not confirm independent third-party reproduction.

(3) The text of standards

For ISO and IEC standards we limited ourselves to number, title, date and scope and did not read the text. We could not confirm an international standard that defines the accuracy of depth sensors, nor one that directly defines how to evaluate a trained recognition model for robotic grasping.

(4) What we did not cover

Perception by touch, the detailed physics of range sensors (LiDAR), estimation of the robot’s own position (SLAM) and perception for autonomous driving are covered in other articles of this series. Models that combine vision and language to output actions are covered in our article on robot foundation models and VLA.

12. How to read the yardsticks and standards

The yardstick for robot vision differs by stage. A range sensor is judged by its error (the scatter about a fitted plane, for instance), detection by IoU and AP, pose by errors in position and orientation, and grasping by the success rate on a real machine. Each yardstick does not guarantee the results of another stage. For safety functions such as “detecting a person” there is a separate safety-standards framework that is distinct from AI recognition. The table lists what we could confirm (checked 25 October 2026).

The arrangement of the table is Our calculation (organising). Numbers, titles, dates and stages of ISO and IEC standards were checked on the official pages (for IEC 61496-3, the IEC scope text shown on the page of the distributor AFNOR). The standards’ text is paid, so we stay with the scope statements.

Notes on reading (for materials engineers)

• Demonstrations, benchmarks and the shop floor are different things: the results of paper demos and public datasets are values for tidy lighting and chosen objects. Re-measure with your own parts, your own lighting and your own dirt
• Specified values and measured values are different: a camera’s distance accuracy and processing speed may be the manufacturer’s specification or a measurement in a paper; separate the two and check the conditions of distance, resolution and surface texture
• Is a standard a requirement, information or a draft? ISO 10218 sets safety requirements for industrial robots, ISO/IEC TR 5469 is a technical report that provides information, and NIST SP 1227 is a draft. “Conforms to a standard” does not mean learned perception is safe
• The scope of certification and approval: protective devices that detect people are assessed within the scope the standard defines (type of device, indoor or outdoor, what is detected). Outside that scope the result does not simply carry over

13. Glossary

14. References

1. Keselman L. et al. “Intel RealSense Stereoscopic Depth Cameras”, arXiv:1705.05548 (2017) — arxiv.org/abs/1705.05548
2. Qi C. R. et al. “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, arXiv:1612.00593 (2016; CVPR 2017) — arxiv.org/abs/1612.00593
3. Besl P. J., McKay N. D. “A method for registration of 3-D shapes”, IEEE Trans. Pattern Anal. Mach. Intell. 14, 239–256 (1992), DOI 10.1109/34.121791 — doi.org/10.1109/34.121791
4. Redmon J. et al. “You Only Look Once: Unified, Real-Time Object Detection”, arXiv:1506.02640 (2015) — arxiv.org/abs/1506.02640
5. He K. et al. “Mask R-CNN”, arXiv:1703.06870 (2017) — arxiv.org/abs/1703.06870
6. Carion N. et al. “End-to-End Object Detection with Transformers”, arXiv:2005.12872 (2020) — arxiv.org/abs/2005.12872
7. Kirillov A. et al. “Segment Anything”, arXiv:2304.02643 (2023) — arxiv.org/abs/2304.02643
8. Ravi N. et al. “SAM 2: Segment Anything in Images and Videos”, arXiv:2408.00714 (2024) — arxiv.org/abs/2408.00714
9. Liu S. et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection”, arXiv:2303.05499 (2023) — arxiv.org/abs/2303.05499
10. Lin T.-Y. et al. “Microsoft COCO: Common Objects in Context”, arXiv:1405.0312 (2014; ECCV 2014) — arxiv.org/abs/1405.0312
11. COCO Consortium “Detection Evaluation” (the metrics used by COCO). Checked 25 October 2026 — cocodataset.org/#detection-eval
12. Xiang Y. et al. “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”, arXiv:1711.00199 (2017) — arxiv.org/abs/1711.00199
13. Calli B. et al. “Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols”, arXiv:1502.03143 (2015) — arxiv.org/abs/1502.03143
14. Hodan T. et al. “BOP: Benchmark for 6D Object Pose Estimation”, arXiv:1808.08319 (2018; ECCV 2018) — arxiv.org/abs/1808.08319
15. Wen B. et al. “FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects”, arXiv:2312.08344 (2023) — arxiv.org/abs/2312.08344
16. Mahler J. et al. “Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics”, arXiv:1703.09312 (2017; RSS 2017) — arxiv.org/abs/1703.09312
17. Sundermeyer M. et al. “Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes”, arXiv:2103.14127 (2021) — arxiv.org/abs/2103.14127
18. Sajjan S. et al. “ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation”, arXiv:1910.02550 (2019) — arxiv.org/abs/1910.02550
19. Yang L. et al. “Depth Anything V2”, arXiv:2406.09414 (2024) — arxiv.org/abs/2406.09414
20. Falco J., Van Wyk K., Messina E. “Performance Metrics and Test Methods for Robotic Hands (Draft)”, NIST Special Publication 1227 (draft), 2018 — doi.org/10.6028/NIST.SP.1227-draft
21. ISO ISO 18646-3:2021 “Robotics — Performance criteria and related test methods for service robots — Part 3: Manipulation” (Edition 1, June 2021). Checked 25 October 2026 — iso.org/standard/73591.html
22. IEC IEC 61496-3:2025 “Safety of machinery — Electro-sensitive protective equipment — Part 3: Particular requirements for active opto-electronic protective devices responsive to diffuse reflection (AOPDDR)” (August 2025; scope text as shown by the distributor AFNOR). Checked 25 October 2026 — boutique.afnor.org (IEC 61496-3:2025)
23. ISO ISO 10218-1:2025 “Robotics — Safety requirements — Part 1: Industrial robots” (Edition 3, February 2025). Checked 25 October 2026 — iso.org/standard/73933.html
24. ISO ISO 10218-2:2025 “Robotics — Safety requirements — Part 2: Industrial robot applications and robot cells” (Edition 2, February 2025). Checked 25 October 2026 — iso.org/standard/73934.html
25. ISO/IEC ISO/IEC TR 5469:2024 “Artificial intelligence — Functional safety and AI systems” (Edition 1, January 2024). Checked 25 October 2026 — iso.org/standard/81283.html
26. ISO/IEC ISO/IEC TS 4213:2022 “Information technology — Artificial intelligence — Assessment of machine learning classification performance” (October 2022). Checked 25 October 2026 — iso.org/standard/79799.html
27. ISO ISO 8373:2021 “Robotics — Vocabulary” (Edition 3, November 2021). Checked 25 October 2026 — iso.org/standard/75539.html

15. Claim-to-source audit

Last updated: 25 October 2026 / Sources are limited to primary material (arXiv papers, the COCO evaluation page, and the official NIST, ISO and IEC pages). We could not read the text of the ISO and IEC standards because it is paid. The numerical examples are calculations from values we assumed and are not the performance of any real camera or robot. We could not confirm, and so do not state, the accuracy of depth sensors as products, statistics on failure rates of image recognition on real machines, or a standard that directly defines the safety assessment of learned perception. Figures 1 and 8 are ImageGen-generated conceptual illustrations; the other figures are explanatory concept diagrams or figures containing our calculations.