Shape-Shifting Splats: Realtime Context Translation for Gaussian Splatting in VR

3D Gaussian Splatting (3DGS) has revolutionized novel-view synthesis by producing photorealistic renderings of virtually any object. However, 3DGS methods primarily support static scene representations, limiting flexibility and preventing direct application in scenarios that require visual variety. Our novel approach, Shape-Shifting Splats, addresses this limitation by enabling dynamic runtime transformations of Gaussian splats. Our versatile formulation allows changes in both color and geometry during render time, without increasing memory usage or introducing visual artifacts. Once trained, our Shape-Shifting Splats generate high-fidelity novel views while supporting modifications in geometry and appearance of the underlying model. In addition to applications in gaming and e-commerce, our method is applicable to mobile, industrial maintenance, and inspection. With multimodal data (e.g., thermal and RGB), we create interactive 3D representations that allow inspectors wearing VR glasses to seamlessly switch between thermal and RGB views, helping to prevent oversights.