CVPR Findings2026
SGST-Transformer: A Spherical Geometry-Aware Spatio-Temporal Transformer for 360° Video Saliency Prediction
Kao Zhang, Tao Song, Zhihua Hu, Ming Li, Xin Ding
Abstract
Panoramic videos have gradually gained wider attention because of the unique sensory experience brought by immersive technology. As a result, the research value of 360° video saliency prediction has become increasingly prominent. However, the unique spherical data format and complex scene content of 360° videos present considerable challenges in developing suitable projection methods for model input and in achieving accurate saliency prediction. Therefore, we propose a Transformer-based model for 360° video saliency prediction. The model adopts a spherical patch projection designed for Transformer, which mitigates geometric distortion through adaptive partitioning and local projection of the spherical surface. Furthermore, it constructs a spatio-temporal feature extraction network. Spatially, it is built upon the Vision Transformer (ViT) architecture and incorporates spherical positional encoding to enhance the perception of spherical geometric structures. Temporally, it introduces a lightweight temporal Transformer for efficient modeling of dynamic video features. Considering the importance of prior knowledge in video saliency prediction, we design a prior tailored to the attention distribution characteristics of 360° videos. It models the visual bias of viewers when watching panoramic videos. Experiments on four benchmark datasets for panoramic video saliency prediction demonstrate that our model outperforms various state-of-the-art methods. The code is available at https://github. com/zhangkao/3DIP-SGST.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction cites TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
Builds on7
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency DetectionKyle Min, Jason J. CorsoICCV 2019 · 189 citations
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense PredictionsChunlong Xia, Xinliang Wang, Feng Lv, Xin Hao et al.CVPR 2024 · 109 citations
- SalientVR: saliency-driven mobile 360-degree video streaming with gaze informationShibo Wang, Shusen Yang, Hailiang Li, Xiaodan Zhang et al.MobiCom 2022 · 43 citations
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency PredictionJunwen Xiong, Peng Zhang, Tao You, Chuanyue Li et al.CVPR 2024 · 12 citations
Related papers
- Distortion-aware Transformer in 360° Salient Object DetectionYinjie Zhao, Lichen Zhao, Qian Yu, Lu Sheng et al.ACM MM 2023 · 14 citations
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion ModelsZixun Fang, Kai Zhu, Zhiheng Liu, Yu Liu et al.NeurIPS 2025 · 2 citations
- STAR-VP: Improving Long-term Viewport Prediction in 360° Videos via Space-aligned and Time-varying FusionBaoqi Gao, Daoxu Sheng, Lei Zhang, Qi Qi et al.ACM MM 2024 · 9 citations
- SphereUFormer: A U-Shaped Transformer for Spherical 360 PerceptionYaniv Benny, Lior WolfCVPR 2025
- ScanTD: 360° Scanpath Prediction based on Time-Series DiffusionYujia Wang, Fang-Lue Zhang, Neil A. DodgsonACM MM 2024 · 11 citations
