Machine vision and SLAM

Before a robot can do anything useful it has to answer two questions: what is around me, and where am I. Neither is given. A robot indoors has no satellite fix, and even outdoors a satellite fix is accurate to metres when the task needs centimetres.

The sensors, and what each is honest about

RGB cameras are cheap, high resolution and rich in meaning: colour, texture, text, human posture, the difference between a shadow and a hole. What a single camera cannot do is measure distance. It infers it, from motion, from perspective, or from a model’s learned expectations, and inference fails on blank walls, glass and unusual objects.

Stereo cameras use two lenses and triangulate, which recovers real depth over a limited range and struggles on surfaces without texture.

Depth cameras project a pattern of infrared light and read the deformation. Excellent indoors, close range, hopeless in sunlight.

Lidar fires laser pulses and times the return, producing a direct distance measurement millions of times a second in a 360-degree field. It gives geometry with no inference at all, works in the dark, does not care about texture, and costs more than the alternatives while struggling in heavy rain, fog and spray.

Radar sees through weather and measures closing speed directly, at low spatial resolution.

Inertial measurement units report acceleration and rotation hundreds of times a second, which fills the gaps between slower sensor readings and drifts badly on their own.

Force and tactile sensing is the one that gets left off lists and does the work in manipulation. A camera cannot tell you how firmly a thing is held; contact can. It is what let Amazon’s Vulcan pick and stow roughly 75 per cent of the item types in a fulfilment centre.

SLAM: two problems that only solve together

Simultaneous localisation and mapping is the name for the chicken-and-egg problem at the centre of mobile robotics. To build a map you need to know where you are. To know where you are you need a map. SLAM does both at once, incrementally.

The SLAM loop, and the error that eats it

The enemy is drift. Every step of estimation carries a small error, and errors compound, so a robot that has walked a long corridor and turned four corners believes it is somewhere it is not, and its map has bent to match. Loop closure is the correction: recognise a previously seen place, work out how far the estimate has slipped, and redistribute the correction back through the whole map.

The variants differ in practice. Visual SLAM uses cameras and is cheap and fragile in the dark. Lidar SLAM is robust and expensive. Visual-inertial odometry fuses cameras with an IMU and is what keeps a drone stable indoors. Semantic SLAM labels the map with what things are rather than only where they are, so a robot’s map contains “door” and “table” instead of a cloud of points, which is what lets an instruction like “put it on the table” resolve to a coordinate.

Where perception still fails

Transparent and reflective surfaces defeat almost everything: glass is invisible to a camera and a mirror to lidar. Weather degrades optical sensing in ways that are hard to detect from inside the system. Repetitive environments, identical aisles, identical corridors, break loop closure by making two different places look the same. And a scene that changes between visits, which is every real workplace, means a stored map is a hypothesis rather than a fact.

Why this decides what robots can do

Every capability higher up depends on this layer being right. A foundation model that plans a task is planning against what perception reported. A self-driving system that yields to a pedestrian first has to have seen one. A warehouse robot that avoids a person is trusting a sensor.

When robots fail in ways that look stupid, the cause is usually here rather than in the reasoning: the machine did exactly what it should have done about a world it had misread.