Panoptic Lifting for 3D Scene Understanding with Neural Fields

Figure 1 . Given only RGB images of an in-the-wild scene as input, our method optimizes a panoptic radiance field which can be queried for color, depth, semantics, and instances for any point in space. We obtain poses for input images with COLMAP [34], and 2D panoptic segmentation masks using a pretrained off-the-shelf network [6] . During training, our method lifts these 2D segmentation masks, which are often noisy and view-inconsistent, into a consistent 3D panoptic radiance field. Once trained, our model is able to render images and their corresponding panoptic segmentation masks from both existing and novel viewpoints.