Decoupled Motion Prediction for Real-time G-buffer Free Frame Extrapolation

Frame extrapolation, as a typical low-latency frame generation method, improves the frame rate of real-time rendering by predicting future frames based solely on historical data. To guarantee the high quality of predictions, existing methods rely heavily on G-buffers of the target frames. However, these G-buffers are not always accessible, and enabling them in certain rendering engines can incur considerable costs. To tackle this challenge, we introduce a G-buffer free frame extrapolation framework that can achieve comparable quality with state-of-the-art G-buffer based methods. In contrast to existing learning-based approaches that handle motions of new frames implicitly and jointly, we design a decoupled strategy that predicts explicit motions for geometry, shading and disoccluded regions separately. In our framework, we first extract the geometric motion using a dual-space method, and then leverage a lightweight motion inpainting network (OccNet) to fill in the disoccluded regions. The shading motion is extracted between two historical frames and then used to propagate shading variations to new frames. Through extensive experiments across various scenes, we demonstrate that our decoupled approach can generate high-quality motions for a wide range of geometric and shading variations in a scene, thereby significantly improving the accuracy of extrapolated frames at a very low computational expense.