Lune

Abstract

We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial-and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks. To facilitate further research, we release code at https://github.com/google-research/scenic .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

536 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training cites Video Swin Transformer
  • VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training cites Language Models are Few-Shot Learners
  • VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training cites Training data-efficient image transformers & distillation through attention
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Segmenter: Transformer for Semantic Segmentation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Video Swin Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Video Swin Transformer cites Training data-efficient image transformers & distillation through attention
  • Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions cites Training data-efficient image transformers & distillation through attention
  • Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer V2: Scaling Up Capacity and Resolution cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer V2: Scaling Up Capacity and Resolution cites Language Models are Few-Shot Learners
  • Swin Transformer V2: Scaling Up Capacity and Resolution cites Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
  • Swin Transformer V2: Scaling Up Capacity and Resolution cites Training data-efficient image transformers & distillation through attention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers536

  1. Video Diffusion ModelsJonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan et al.NeurIPS 2022 · 2,948 citations
  2. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-TrainingZhan Tong, Yibing Song, Jue Wang, Limin WangNeurIPS 2022 · 2,336 citations
  3. Swin Transformer V2: Scaling Up Capacity and ResolutionZe Liu, Han Hu, Yutong Lin, Zhuliang Yao et al.CVPR 2022 · 2,138 citations
  4. Segmenter: Transformer for Semantic SegmentationRobin Strudel, Ricardo Garcia, Ivan Laptev, Cordelia SchmidICCV 2021 · 1,898 citations
  5. Video Swin TransformerZe Liu, Jia Ning, Yue Cao, Yixuan Wei et al.CVPR 2022 · 1,847 citations

Builds on28

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
  4. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  5. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without ConvolutionsWenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan et al.ICCV 2021 · 4,909 citations

Related papers

  1. Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius, Heng Wang, Lorenzo TorresaniICML 2021 · 2,927 citations
  2. Can an Image Classifier Suffice For Action Recognition?Quanfu Fan, Chun-Fu Chen, Rameswar PandaICLR 2022 · 39 citations
  3. Multiview Transformers for Video RecognitionShen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu et al.CVPR 2022 · 279 citations
  4. BEVT: BERT Pretraining of Video TransformersRui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen et al.CVPR 2022 · 200 citations
  5. Deformable Video TransformerJue Wang, Lorenzo TorresaniCVPR 2022 · 40 citations
Dusk over the sea between two cliffs drawn in fine vertical lines