Real-Time Object Detection

What Real-Time Means in Object Detection

The threshold for "real-time" depends on the application. Video typically runs at 24 to 60 frames per second, so a detector that processes at least 30 frames per second (33 milliseconds per frame) is considered real-time for most video applications. Autonomous driving requires detection at camera frame rate (30 to 60 fps) with latency under 100 milliseconds, because any delay translates to distance traveled without perception: at highway speeds of 120 km/h, a 100-millisecond delay means the car travels 3.3 meters blind. Augmented reality requires even lower latency, under 20 milliseconds, because humans perceive delays above this threshold as lag between their head movements and the displayed overlay.

Achieving real-time speed is a fundamentally different engineering challenge than achieving maximum accuracy. The most accurate detectors in the research literature often process images in 200 to 500 milliseconds each, which is fine for batch processing of photo archives but far too slow for live video. Real-time detection requires architectural choices that sacrifice some accuracy for dramatic speed improvements, and the history of the field is largely the story of pushing the accuracy-speed trade-off curve to get more accuracy at any given speed budget.

Detection speed is measured in several ways. Frames per second (FPS) on a specific GPU is the most intuitive metric. Latency (milliseconds per frame) is more relevant for interactive applications. FLOPs (floating-point operations) measures the computational cost independent of hardware, useful for comparing architectures across different deployment targets. In practice, FPS on a standardized GPU (typically an NVIDIA V100 or A100) is the most commonly reported metric in research papers.

Two-Stage vs Single-Stage Detectors

The first successful deep learning detector, R-CNN (Regions with CNN features, 2014), used a two-stage approach. The first stage proposes candidate regions that might contain objects using a separate algorithm (originally Selective Search, later a Region Proposal Network). The second stage classifies each proposed region and refines its bounding box. This approach is accurate because the classifier sees a tightly cropped image patch for each candidate and can focus its full capacity on deciding whether that specific patch contains an object. However, processing 300 to 2000 region proposals per image makes two-stage detectors inherently slow. Faster R-CNN (2015) improved speed significantly by sharing convolutional features across proposals but still runs at 5 to 15 FPS on typical hardware, below the real-time threshold for most applications.

Single-stage detectors eliminate the region proposal step entirely, predicting all detections in a single forward pass through the network. SSD (Single Shot MultiBox Detector, 2016) divides the image into a grid and predicts bounding boxes and class scores at each grid cell across multiple feature map scales. This runs at 59 FPS on a GPU while achieving accuracy competitive with Faster R-CNN. RetinaNet (2017) introduced the focal loss function, which addresses the extreme imbalance between background and foreground examples in single-stage detectors by down-weighting the loss contribution of well-classified easy examples. With focal loss, single-stage detectors matched two-stage accuracy for the first time while maintaining their speed advantage.

The accuracy gap between two-stage and single-stage detectors has largely closed in modern architectures. Contemporary single-stage detectors like YOLO variants achieve accuracy equivalent to or exceeding two-stage methods while running at 5 to 20 times the speed. Two-stage detectors remain useful for applications where maximum accuracy justifies slower processing, such as medical image analysis and satellite imagery interpretation, but single-stage architectures dominate all real-time applications.

YOLO: The Defining Real-Time Architecture

YOLO (You Only Look Once), published by Joseph Redmon in 2016, fundamentally reframed object detection as a regression problem. Instead of classifying thousands of region proposals, YOLO divides the image into an S x S grid (originally 7 x 7) and has each grid cell directly predict B bounding boxes with confidence scores and C class probabilities. The entire detection task is a single pass through a convolutional network, making YOLO dramatically faster than anything that came before. The original YOLO processed images at 45 FPS, with a smaller variant reaching 155 FPS, while maintaining reasonable accuracy.

YOLOv2 (2017) introduced batch normalization, anchor boxes (predefined aspect ratios for better shape matching), and multi-scale training. YOLOv3 (2018) adopted a feature pyramid network that detects objects at three different scales, dramatically improving detection of small objects which the original YOLO struggled with. YOLOv3 became the workhorse detector for practical applications, running at 30+ FPS while achieving mAP (mean Average Precision) scores competitive with much slower detectors.

After the original author stepped away from the project, the YOLO lineage continued through community-driven development. YOLOv4 (2020) introduced a bag of training tricks including mosaic data augmentation, self-adversarial training, and cross-stage partial connections. YOLOv5 reimplemented the architecture in PyTorch with a focus on ease of deployment. YOLOv7 and YOLOv8 pushed the accuracy-speed frontier further, with YOLOv8 achieving 53.9% mAP on the COCO benchmark while running at over 100 FPS on an A100 GPU. YOLOv9 and YOLO11 introduced further architectural refinements including programmable gradient information and efficient layer aggregation.

The YOLO family's practical dominance comes not just from raw performance but from the ecosystem surrounding it. Pre-trained weights for dozens of object classes are freely available. Training on custom datasets requires only labeled images and a few lines of configuration. Export to deployment formats (ONNX, TensorRT, CoreML, TFLite) is built in. This combination of performance, simplicity, and deployment flexibility has made YOLO the default choice for anyone building a real-time detection application.

Making Detection Fast: Optimization Techniques

Achieving real-time speed requires optimizing both the neural network architecture and the inference pipeline. Network architecture optimizations include depthwise separable convolutions (which reduce computation by factoring standard convolutions into a spatial filtering step and a channel mixing step, cutting FLOPs by 8 to 9 times), channel pruning (removing filters that contribute least to accuracy), and knowledge distillation (training a small "student" network to mimic the outputs of a larger, more accurate "teacher" network). MobileNet and EfficientNet families were designed specifically for efficient inference, achieving reasonable accuracy with 10 to 100 times fewer parameters than full-size architectures.

Quantization reduces the numerical precision of model weights and activations from 32-bit floating point to 16-bit, 8-bit, or even 4-bit integers. This shrinks model size proportionally and speeds up inference on hardware with integer acceleration units. INT8 quantization typically reduces model size by 4x and improves inference speed by 2 to 3x on GPUs with Tensor Cores, while accuracy drops by less than 1% on standard benchmarks. Post-training quantization applies quantization to a pre-trained model without retraining. Quantization-aware training incorporates simulated quantization during training, producing models that lose even less accuracy when quantized.

TensorRT, NVIDIA's inference optimization library, applies graph-level optimizations including layer fusion (combining multiple sequential operations into a single kernel), precision calibration, and kernel auto-tuning specific to the target GPU architecture. A model optimized with TensorRT typically runs 2 to 5 times faster than the same model run through a general-purpose framework like PyTorch. Similar optimization tools exist for other deployment targets: CoreML for Apple devices, TFLite for Android, and OpenVINO for Intel hardware.

Edge Deployment: Detection on Small Devices

Many real-time detection applications require running on edge devices rather than cloud servers: drones, security cameras, mobile phones, industrial cameras, and embedded systems in vehicles. These devices have dramatically less compute power than data center GPUs, often running on mobile GPUs, neural processing units (NPUs), or even CPUs. A detector that runs at 100 FPS on an A100 GPU might run at 5 FPS on a Jetson Nano or 2 FPS on a Raspberry Pi without optimization.

NVIDIA's Jetson family (Nano, Orin Nano, Orin NX, AGX Orin) are the most popular edge platforms for vision AI, offering 20 to 275 TOPS (trillion operations per second) of INT8 compute in power envelopes from 7 to 60 watts. A YOLOv8-nano model optimized with TensorRT runs at 30+ FPS on a Jetson Orin Nano, sufficient for real-time applications. Google's Coral Edge TPU offers 4 TOPS in a USB stick form factor, running lightweight models like EfficientDet-Lite at 20+ FPS while consuming only 2 watts. Qualcomm's AI Engine in Snapdragon processors enables on-device detection in smartphone cameras, powering features like scene recognition, document scanning, and AR object placement.

Deploying detection on edge devices requires careful model selection and optimization. The model must fit within the device's memory constraints (often 2 to 8 GB shared between CPU and GPU). Inference must complete within the latency budget at the device's compute capability. Power consumption must stay within the thermal envelope. These constraints often require using smaller model variants (YOLOv8-nano instead of YOLOv8-xlarge), reducing input resolution (from 640x640 to 320x320), limiting the number of detection classes, and applying aggressive quantization. The skill of edge deployment lies in finding the optimal trade-off between these parameters for each specific use case.

Beyond Bounding Boxes: Real-Time Instance Segmentation

Object detection produces rectangular bounding boxes, but many applications need the precise shape of each detected object. Instance segmentation assigns a pixel-level mask to each detected object, distinguishing not just what and where but the exact spatial extent. Historically, instance segmentation was too computationally expensive for real-time processing, but recent architectures have brought it within reach.

YOLACT (You Only Look At CoefficienTs, 2019) was the first real-time instance segmentation model, running at 33 FPS by generating a set of prototype masks in parallel with detection and combining them with per-instance coefficients. YOLOv8-seg integrates instance segmentation into the YOLO architecture, producing both bounding boxes and segmentation masks in a single forward pass at speeds above 60 FPS on modern GPUs. Real-time panoptic segmentation, which labels every pixel in the frame as either a specific object instance or a background class, has also become feasible with architectures like Real-Time Panoptic Segmentation (RT-PANO).

Real-time tracking adds temporal continuity to frame-by-frame detection. Algorithms like ByteTrack and BoT-SORT associate detections across frames using motion prediction and appearance matching, maintaining persistent object identities through a video sequence. A tracking system layered on top of a real-time detector adds only 1 to 3 milliseconds of overhead per frame, enabling applications that need to follow specific objects over time: counting people entering a store, tracking vehicles through an intersection, or following a player across a sports broadcast.