Depth Sensing in Practice: What Actually Worked in 2022
I spent most of 2022 debugging depth maps for a warehouse robotics project. The manufacturer provided a sensor that looked great on the spec sheet and performed fine in their showroom. Our warehouse had polished concrete floors, glass shelving, and sunlight streaming through loading bay doors between 2pm and 4pm. The sensor could not distinguish between a real pallet and its reflection on the floor about sixty percent of the time. We ended up running a stereo rig alongside the time-of-flight sensor and blending the outputs. It added complexity but the fallback held up. If you are building something with depth data, assume the hardware will fail on a specific edge case and plan for it before you need it. The year was defined by two things. The first was the maturation of stereo depth sensors for consumer applications. The second was the continued push to replace dedicated depth hardware with software-based approaches using standard RGB cameras. Apple's LiDAR scanners appeared on more devices than ever. Intel's RealSense line was winding down after the company exited the division in early 2022, which sent a ripple through the developer community. Microsoft's Project Malmo and related research continued to use depth-aware environments for reinforcement learning agents. On the software side, models like Depth Anything, MAE-based depth estimation, and improvements to MiDaS produced usable single-image depth maps without any specialized hardware. The core tension in the field remained the same as it has been since structured light showed up in the original Kinect. Accuracy, cost, range, and working in direct sunlight are still competing constraints. You can usually optimize for three of them. The fourth will bite you later.
The Hardware Side of Things
Time-of-Flight Sensors
ToF sensors measure the phase shift of reflected infrared light to calculate distance. They are fast and produce dense depth maps at high frame rates. The problem is specular surfaces. A mirror, a wet floor, or a shiny plastic bin will scatter or absorb the IR signal in ways that produce garbage data. I worked with the Sony DepthSense module and the Intel D435i before the latter became difficult to source. The D435i uses stereo IR pairs and is fundamentally more robust to reflective surfaces than pure ToF, but it still struggles with low-texture walls. If your environment has large stretches of uniform color, add texture or use a structured light pattern if your sensor supports it. Stereo depth works by finding corresponding points in two offset camera images. It requires texture. No texture means no matches. But it also works in direct sunlight because it does not emit its own illumination. For outdoor robotics, stereo is usually the safer choice. The OAK-D line from Luxonis became popular because it integrates inference on the chip and provides depth out of the box with reasonable performance. The catch is baseline selection. A narrow baseline gives you better accuracy at close range but limits maximum depth. A wide baseline does the opposite. Most users pick a sensor and then discover too late that the fixed baseline does not match their operating range. Structured light projects a known pattern and measures distortion. It is accurate at short range and immune to some of the ambiguities that plague stereo. The iPhone Face ID system is a mature example. The downside is ambient light sensitivity. Bring a structured light sensor outside on a bright day and the projected pattern gets washed out. It works fine indoors. It does not work well near windows.
This is where 2022 got interesting. Single-image depth estimation models had crossed a threshold where they were actually usable without calibration. MiDaS v3, Zero123 variants, and especially Depth Anything produced surprisingly coherent results from a single RGB frame. The models are trained on large datasets combining synthetic data, sparse LiDAR scans, and stereo-derived ground truth. They generalize reasonably well but they have blind spots. The biggest one is scale ambiguity. A model will tell you that object A is closer than object B, but it cannot reliably tell you that A is three meters away and B is five meters away unless you provide a reference. This matters more than people admit. If you are using monocular depth for navigation, you need a scale recovery step. Common approaches include fusing with IMU data, using known object sizes, or applying a simple linear scaling based on a single measured point. None of these are perfect. All of them are better than nothing. Another issue is temporal consistency. Running a frame-by-frame model produces jittery depth maps because each frame is processed independently. Adding a temporal smoothing filter or using a recurrent architecture helps, but introduces lag. In a real-time system, lag is its own failure mode. I found that a simple exponential moving average with a per-pixel alpha tuned to the expected scene dynamics worked well enough for our use case without adding significant latency.
Get the Full Details
Practical Tips That Are Not Obvious
Calibration matters more than you think. The extrinsic calibration between your RGB camera and your depth sensor degrades over time if the mount flexes or vibrates. Check it monthly if the device moves. A drift of even a few millimeters at the mount can shift the depth map by centimeters at range, which is enough to cause collisions in close-quarters robotics. Irregular sampling is a real problem with ToF. Pixels near the edge of the field of view often have lower confidence values that the sensor does not always expose properly. I learned this the hard way when a robot drove toward a doorway and the depth map showed empty space along the edges. The doorway frame was being misread as open space. Masking the peripheral region and relying on central pixels only cut the usable field of view but eliminated the failures. Filtering depth data is not just about removing noise. Median filters remove outlier points but also blur object boundaries. Bilateral filtering preserves edges but can create halos around high-contrast boundaries. For object segmentation workflows, I recommend a guided filter using the RGB image as the guide. It smooths depth while respecting edges in the color image. The OpenCV implementation is fast and runs in real time on most GPUs.
Common Pitfalls
The most common mistake I see is treating depth data as ground truth. It is never ground truth. Every depth sensor has a confidence metric or variance estimate baked into its output. Use it. Pixels flagged as low confidence should be treated as missing data, not as correct measurements with uncertainty. Ignoring the confidence channel is why depth systems sometimes appear to work perfectly in demos and fail dramatically in production. Another mistake is assuming that better hardware solves data quality problems. It does not. A more expensive sensor will give you cleaner data under ideal conditions. It will still fail on the same fundamental edge cases. Budget for data collection and validation in your environment before you trust the numbers.
Where the Field Is Headed
The gap between hardware-based and software-based depth is narrowing. Models trained on massive datasets are approaching the accuracy of mid-range stereo rigs in controlled settings. At the same time, new sensor designs like event-based depth cameras and FMCW LiDAR for consumer devices are beginning to appear. The constraints will shift rather than disappear. You will trade calibration effort for compute cost, or range accuracy for power consumption. The decisions are practical, not theoretical. If you are starting a project now, the most pragmatic path is a stereo camera with a known baseline, calibrated regularly, combined with a temporal smoothing layer and a confidence mask. Fall back to monocular depth estimation only when you have validated that the scale recovery works for your specific deployment. Test in your actual environment with your actual lighting conditions before you ship anything.
