What You See is What You Get: Exploiting Visibility for 3D Object Detection

What is a good representation for 3D sensor data? We visualize a bird's-eye-view LiDAR scene and highlight two regions that may contain an object. Many contemporary deep networks process 3D point clouds, making it hard to distinguish the two regions (left). But depth sensors provide more than 3D points -they provide estimates of freespace in between the sensor and the measured 3D point. We visualize freespace by raycasting (right), where green is free and white is unknown. In this paper, we introduce deep 3D networks that leverage freespace to significantly improve 3D object detection accuracy.