Frame Complexity-Aware Foveated Video Encoding for Real-time High-Quality Streaming

VR streaming and VR cloud gaming require high-resolution video streaming to provide users with high quality visual experience and maximize their interaction and immersion. Consequently, the video streams have a high bitrate and require a large amount of available bandwidth. Foveated video encoding (FVE) reduces bandwidth demand by selectively allocating higher quality to perceptually relevant regions based on human visual characteristics. However, scenes with high spatial detail or rapid motion introduce spatial and temporal frame complexities. Conventional video encoding assesses complexity and manages quality and bitrate through an internal rate-control mechanism. However, the SoTA FVE methods perform the quality allocation process after the rate control has already run. This may lead to rate violations and, consequently, under-or overutilization of available bandwidth, which in turn causes increased latency and/or reduced visual quality. In this paper, we present a real-time complexity-adaptive FVE method that minimizes computational latency using a GPU-accelerated compute shader. By prioritizing key spatial and temporal complexity variables, our weighted complexity estimation optimizes quality assignment. Our method outperforms the complexity-agnostic FVE benchmark with 44-78% greater bitrate stability, 21% lower latency, and 7% higher perceptual quality. It also surpasses the partial complexity-aware FVE benchmark, delivering 18-94% better network utilization alongside a 7% latency reduction and a 4-5% gain in quality. Our method also ensures generalizability across diverse scenarios.