Robot Sensors and Perception: How Robots See, Touch, and Navigate

Proprioceptive Sensors: The Robot's Internal Senses

Proprioceptive sensors measure the robot's own internal state, similar to how your body senses the position of your limbs without looking at them. These sensors are essential for control, because a robot cannot correct its movements if it does not know where its joints are or how much force its motors are producing.

Encoders

Encoders are the most fundamental sensors in robotics. They measure the rotational position of a motor shaft or joint. Incremental encoders produce a pulse for each small increment of rotation. A typical incremental encoder produces 1,000 to 10,000 pulses per revolution, and the controller counts these pulses to track position. The limitation is that the count is relative, so if power is lost, the robot does not know its absolute position and must be re-homed (moved to a known reference position).

Absolute encoders output a unique digital code for every angular position, so the robot always knows its exact position even after a power cycle. Multi-turn absolute encoders also track how many full revolutions the shaft has completed. High-end industrial robots use 19-bit to 25-bit absolute encoders, providing angular resolution as fine as 0.001 degrees.

Force and Torque Sensors

Force/torque (F/T) sensors measure the forces and moments (torques) acting on a structure, typically mounted between the robot's wrist and its end effector. A six-axis F/T sensor measures three force components (Fx, Fy, Fz) and three torque components (Tx, Ty, Tz), providing a complete picture of the interaction forces between the robot and whatever it is touching.

F/T sensors enable tasks that require controlled contact: grinding and polishing (maintaining constant pressure against a surface), assembly (detecting when a part is properly seated by the force signature), quality inspection (measuring how much force is needed to push a button or close a latch), and human-robot collaboration (detecting unexpected contact with a person and stopping immediately). ATI Industrial Automation and OnRobot are leading manufacturers of robotic F/T sensors.

Current Sensors

Every electric motor draws current proportional to the torque it produces. By measuring motor current, the robot's controller can estimate the load on each joint without a dedicated F/T sensor. This approach is less accurate than a dedicated sensor but comes essentially for free, since motor drivers already measure current for their own control purposes. Collaborative robots from Universal Robots and FANUC use current sensing as a primary method for detecting unexpected contact with people.

Inertial Measurement Units (IMUs)

IMUs combine accelerometers (which measure linear acceleration) and gyroscopes (which measure rotational velocity) to track a robot's orientation and motion. Three-axis accelerometers and three-axis gyroscopes together provide six-axis inertial data. More advanced IMUs add magnetometers (electronic compasses) for absolute heading reference.

IMUs are critical for mobile robots that need to maintain balance (bipedal walkers, drones) or track their orientation as they move through the world. A drone's flight controller reads IMU data at 1,000 Hz or faster to maintain stable hovering. The Boston Dynamics Atlas humanoid robot uses IMU data combined with joint sensors and ground contact sensors to maintain balance during dynamic maneuvers like running, jumping, and backflipping.

Exteroceptive Sensors: Perceiving the World

Exteroceptive sensors measure the environment outside the robot, providing the data needed for navigation, object recognition, obstacle avoidance, and task execution.

Cameras

Cameras are the most versatile sensors in robotics, providing rich visual information about the environment. Monocular cameras (single lens) capture 2D images and are the cheapest option. With deep learning, a single camera can detect and classify objects, read text, estimate distances, and track motion. However, a single camera cannot directly measure depth without additional processing or assumptions.

Stereo cameras use two lenses separated by a known distance (the baseline) to calculate depth through triangulation, the same principle your eyes use to perceive depth. Intel RealSense and ZED cameras are popular stereo vision systems in robotics, providing both RGB color images and depth maps at frame rates of 30 to 90 Hz.

Depth cameras use structured light (projecting a known pattern of dots or lines and analyzing how the pattern deforms on surfaces) or time-of-flight (measuring how long light takes to bounce back from surfaces) to measure depth directly. Microsoft's Azure Kinect and Intel RealSense D400 series are widely used depth cameras. They produce dense point clouds with sub-centimeter accuracy at ranges up to 10 meters.

Event cameras are a newer technology that reports individual pixel changes rather than capturing full frames. Each pixel independently fires when it detects a change in brightness, producing a stream of events with microsecond timing resolution. Event cameras excel at detecting fast motion and operating in challenging lighting conditions (very bright or very dark scenes). They are gaining traction in drone and autonomous vehicle applications.

LiDAR

LiDAR (Light Detection and Ranging) measures distances by sending out laser pulses and measuring the time they take to reflect back. A spinning LiDAR sensor sweeps a laser beam around 360 degrees, building a detailed 3D point cloud of the surroundings at rates of 10 to 20 revolutions per second. Each revolution produces hundreds of thousands of distance measurements.

LiDAR is the primary sensor for autonomous vehicle navigation, warehouse robot mapping, and any application requiring accurate, long-range 3D measurements. Velodyne, Ouster, and Luminar are major LiDAR manufacturers. Solid-state LiDAR, which eliminates the spinning mechanism in favor of a fixed semiconductor emitter, is rapidly dropping in price and is expected to cost under $100 per unit at mass production volumes by 2027.

Ultrasonic Sensors

Ultrasonic sensors emit high-frequency sound pulses and measure the echo return time to calculate distance. They are inexpensive ($2 to $20), simple to interface, and work well for detecting objects at ranges of 2 centimeters to 5 meters. They are widely used in hobby robots, automotive parking sensors, and simple obstacle avoidance systems. Their limitations include low angular resolution (the sound beam is wide, so you cannot pinpoint exactly where an object is) and poor performance on soft, sound-absorbing surfaces like fabric or foam.

Tactile Sensors

Tactile sensors detect physical contact, pressure, and surface texture. Resistive tactile sensors change electrical resistance when pressed. Capacitive sensors detect changes in capacitance caused by proximity or contact. Piezoelectric sensors generate voltage when deformed, useful for detecting dynamic touch events like vibration or slip.

Tactile sensing is essential for dexterous manipulation, where a robot must grip delicate objects without crushing them, detect when an object is slipping from its grasp, and feel the difference between materials. GelSight and DIGIT sensors use cameras behind a flexible gel pad to produce high-resolution tactile images that can resolve surface features as small as 10 micrometers.

Proximity Sensors

Proximity sensors detect the presence of nearby objects without physical contact. Infrared proximity sensors emit an IR beam and measure the reflected light intensity, useful for short-range detection (1 to 80 cm). Capacitive proximity sensors detect nearby conductive or dielectric materials through changes in an electric field. Magnetic proximity sensors (Hall effect sensors) detect nearby magnets or ferromagnetic materials. These sensors are commonly used for limit detection (knowing when a joint has reached its end of travel), part presence detection, and simple obstacle avoidance.

Sensor Fusion

No single sensor provides a complete picture of the world. Cameras provide rich visual detail but struggle in darkness and cannot directly measure force. LiDAR measures distances precisely but cannot read text or distinguish colors. Force sensors detect contact but know nothing about objects the robot has not yet touched. Sensor fusion combines data from multiple sensors to create a representation of the world that is more accurate and complete than any single sensor could provide.

The most common approach to sensor fusion is the Kalman filter, a mathematical algorithm that combines noisy measurements from multiple sources into an optimal estimate of the system's state. An autonomous vehicle's Kalman filter might combine GPS position (accurate to 2 meters, updated at 10 Hz), IMU acceleration (noisy but updated at 1,000 Hz), wheel odometry (subject to slip), and LiDAR-based localization (accurate to centimeters but computationally expensive) into a single position estimate that is more accurate than any individual source.

More advanced fusion methods include particle filters (which represent the state as a cloud of hypothetical particles, useful for non-linear systems), graph-based optimization (which treats the fusion problem as a graph where nodes are states at different times and edges are sensor measurements), and deep learning fusion (which trains neural networks to combine camera, LiDAR, and radar data end-to-end).

Computer Vision for Robots

Computer vision is the AI subfield dedicated to enabling machines to understand images and video. For robots, the key computer vision tasks are:

Object detection identifies what objects are in the scene and where they are, drawing bounding boxes around each detected object. Models like YOLO (You Only Look Once) and Detectron2 can detect dozens of object classes in real time on embedded hardware.

Object pose estimation determines not just where an object is but how it is oriented in 3D space. This is essential for grasping, because a robot needs to know both the position and the rotation of an object to plan a successful grip.

Semantic segmentation classifies every pixel in an image into a category (floor, wall, table, person, robot arm), enabling the robot to understand the full layout of its environment.

Visual SLAM (Simultaneous Localization and Mapping) uses camera data to build a map of the environment while simultaneously tracking the robot's position within that map. ORB-SLAM3 and RTAB-Map are popular visual SLAM implementations used in mobile robots.

Deep learning has transformed computer vision in robotics. Before deep learning, engineers hand-crafted feature detectors and classifiers for each specific application. Now, convolutional neural networks and vision transformers learn features automatically from training data, achieving human-level or better performance on many perception tasks. The challenge has shifted from designing algorithms to collecting and labeling sufficient training data.