SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction

Only Video Sequences As Supervision Self-Supervised 3D Occupancy Prediction Semantics Geometry lower higher …… SelfOcc Optional 2D Segmentor car driveable surface sidewalk terrain manmade vegetation truck Figure 1 . Trained with only video sequences as supervision, our model can predict meaningful geometry for the scene given surroundcamera RGB images, which can be further extended to semantic occupancy prediction if 2D segmentation maps are available e.g. from an off-the-shelf segmentor. This task is challenging because it completely depends on video sequences to reconstruct scenes without any 3D supervision. We observe that our model can produce dense and consistent occupancy prediction and even infer the back side of cars.