Interactive Mixed Reality Rendering on Holographic Pyramid

Currently, ray tracing and image-based lighting (IBL) have shortcomings when rendering the metallic virtual object displayed in the holographic pyramid in mixed reality. Ray tracing can hardly achieve the interactive frame rates, and IBL cannot accurately render the reflection result of the foreground near the virtual object. In this paper, we propose a mixed reality rendering method to render glossy and specular reflection effects on metallic virtual objects displayed in the holographic pyramid based on the surrounding real environment at interactive frame rates. First, we acquire the real environment data with four RGBD cameras and a panoramic camera; then, we introduce a foreground point cloud generation method to extract a temporally stable foreground point cloud from RGBD videos captured in real time; after this, we propose an efficient ray tracing method to render the dynamic glossy and specular reflections on the virtual objects that are displayed in the holographic pyramid. We test our method on several real and synthetic scenes. Compared with IBL and screen-space ray tracing, our method can generate the rendering results closer to the ground truth at the same time cost. Compared with Monte Carlo path tracing, our method is 2.5-4.5× faster in generating rendering results of the comparable quality.