3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting

https://research.nvidia.com/labs/toronto-ai/3DGUT Ground Truth Trained w/ Undistorted Views Trained w/ Original Views Secondary Effects Refractions Reflections Figure 1. We extend 3D Gaussian Splatting (3DGS) to support nonlinear camera projections and secondary rays for simulating phenomena such as reflections and refractions. By replacing EWA splatting rasterization with the Unscented Transform, our approach retains real-time efficiency while accommodating complex camera effects like rolling shutter. (Left) A comparison of our model trained on undistorted views vs. the original distorted fisheye views, showing that training on the full set of pixels improves visual quality. (Right) Two synthetic objects, a reflective sphere and a refractive statue, inserted into a scene reconstructed with our model.