Lune

CVPR2022Top-tier venue

Multiview Transformers for Video Recognition

Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, Cordelia Schmid

Abstract

Video understanding requires reasoning at multiple spatiotemporal resolutions – from short fine-grained motions to events taking place over longer durations. Although transformer architectures have recently advanced the state-of-the-art, they have not explicitly modelled different spatiotemporal resolutions. To this end, we present Multiview Transformers for Video Recognition (MTV). Our model consists of separate encoders to represent different views of the input video with lateral connections to fuse information across views. We present thorough ablation studies of our model and show that MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost across a range of model sizes. Furthermore, we achieve state-of-the-art results on six standard datasets, and improve even further with large-scale pretraining. Code and checkpoints are available at: https://github.com/google-research/scenic.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

74 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
  • HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions cites Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
  • HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions cites Training data-efficient image transformers & distillation through attention
  • ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
  • ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning cites Language Models are Few-Shot Learners
  • Earthformer: Exploring Space-Time Transformers for Earth System Forecasting cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Training data-efficient image transformers & distillation through attention
  • Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions cites Training data-efficient image transformers & distillation through attention
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
  • Masked Autoencoders As Spatiotemporal Learners cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
  • Masked Autoencoders As Spatiotemporal Learners cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers74

  1. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  2. Masked Autoencoders As Spatiotemporal LearnersChristoph Feichtenhofer, Haoqi Fan, Yanghao Li, Kaiming HeNeurIPS 2022 · 690 citations
  3. HorNet: Efficient High-Order Spatial Interactions with Recursive Gated ConvolutionsYongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou et al.NeurIPS 2022 · 422 citations
  4. Earthformer: Exploring Space-Time Transformers for Earth System ForecastingZhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu et al.NeurIPS 2022 · 410 citations
  5. ST-Adapter: Parameter-Efficient Image-to-Video Transfer LearningJunting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao et al.NeurIPS 2022 · 290 citations

Builds on23

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  3. Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
  4. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without ConvolutionsWenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan et al.ICCV 2021 · 4,909 citations
  5. RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations

Related papers

  1. ViViT: A Video Vision TransformerAnurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun et al.ICCV 2021 · 2,947 citations
  2. Multiscale Vision TransformersHaoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li et al.ICCV 2021 · 1,611 citations
  3. Compositional Video Understanding with Spatiotemporal Structure-based TransformersHoyeoung Yun, Jinwoo Ahn, Minseo Kim, Eun-Sol KimCVPR 2024 · 4 citations
  4. Video Swin TransformerZe Liu, Jia Ning, Yue Cao, Yixuan Wei et al.CVPR 2022 · 1,847 citations
  5. Mavors: Multi-granularity Video Representation for Multimodal Large Language ModelYang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu et al.ACM MM 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines