SalientVR: saliency-driven mobile 360-degree video streaming with gaze information

Mobile 360° video streaming has grown significantly in popularity but the quality of experience (QoE) suffers from insufficient wireless network bandwidth. The state-of-the-art solutions are limited by the temporal correlation assumption. Recent studies are aware of the potential of saliency to further QoE improvement, but several fundamental challenges about saliency judgment, saliency acquirement, and quality adaptation are still not fully addressed. To solve these challenges, we present SalientVR, a saliency-driven mobile 360° video streaming system integrated with gaze information. We design (i) a precise gaze-driven saliency judging criterion for mobile VR viewers, (ii) two pragmatic gaze-driven, tile-level saliency acquiring methods based on cross-user similarity and a specific content-aware deep neural network respectively, and (iii) a lightweight saliency-aware quality adaptation algorithm with a motion-assisted online correction, which is robust to wireless bandwidth vagaries and saliency bias. Moreover, we contribute a gaze-annotated dataset and a gaze-driven quality assessment metric for 360° videos. By extensive prototype evaluations (based on dataset tests and user studies), compared to alternatives, SalientVR significantly enhances the video quality and reduces the rebuffering ratio over 4G/LTE network emulations and in the wild, which achieves a 43.68% QoE improvement.