Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming

Virtual Reality (VR) and Augmented Reality (AR) technologies have become popular in recent years. Encoding and transmitting the omni-directional or $360^\circ$ video is critical and challenging for those applications. The $360^\circ$ video requires much higher bandwidth than the traditional planar video. A premium quality $360^\circ$ video with 120 frames per second (fps) and 24K resolution can easily consume bandwidth in the range of Gigabits-per-second 1. On the other hand, at any given time, a user only watches a small portion of the $360^\circ$ scope within her Field-of-View (FoV). An effective way to reduce the bandwidth requirement of $360^\circ$ video is through FoV-adaptive streaming, which codes and delivers the predicted FoV region at higher quality, and discards or codes at lower quality the remaining regions. Such strategy has been quite extensively studied for video-on-demand _adapt_2,fov_adapt_3,1,tile_based_3,qian2016optimizing and live video streaming applications_1,live_2,live_3, sun2020flocking. Interactive applications, such as conferencing, gaming, and remote collaboration, can also benefit from $360^\circ$ video by creating an immersive environment for participants to interact with each other citeinteractive_gamming _conferencing 2015outatime. However, realtime coding and streaming of $360^\circ$ video with extremely low latency, required for interactive applications, has not been sufficiently addressed. This work focuses on developing low-latency and FoV-adaptive coding and streaming strategies for interactive $360^\circ$ video streaming. We assume the sender and the receiver are connected by a network path with dynamically varying throughput without short-latency guarantee. The sender is either the video source, or a proxy server relaying the source video. The receiver is either the end user device that directly renders the video, or a local edge server that renders the video and transmit to the end user 2017.