Lune

ICML2021Top-tier venue

OmniNet: Omnidirectional Representations from Transformers

Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Prakash Gupta, Philip Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, Donald Metzler

Abstract

This paper proposes Omnidirectional Representations from Transformers (OMNINET). In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network. This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al., 2020), low-rank attention (Wang et al., 2020) and/or Big Bird (Zaheer et al., 2020) as the meta-learner. Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena (LRA), and Image Recognition. The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B, WMT'14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation in Vision Transformers leads to significant improvements on image recognition tasks on both few-shot learning and fine-tuning setups.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Long Range Arena : A Benchmark for Efficient Transformers cites Big Bird: Transformers for Longer Sequences
  • Continuous Diffusion Model for Language Modeling cites Simple and Effective Masked Diffusion Language Models
  • Long-Short Transformer: Efficient Transformers for Language and Vision cites Long Range Arena : A Benchmark for Efficient Transformers
  • Long-Short Transformer: Efficient Transformers for Language and Vision cites Big Bird: Transformers for Longer Sequences
  • Long-Short Transformer: Efficient Transformers for Language and Vision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Long-Short Transformer: Efficient Transformers for Language and Vision cites Language Models are Few-Shot Learners
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Generative Pretraining From Pixels
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Exploring the Limits of Large Scale Pre-training cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Exploring the Limits of Large Scale Pre-training cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. Simple and Effective Masked Diffusion Language ModelsSubham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan et al.NeurIPS 2024 · 929 citations
  2. Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained TransformersLirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming HeNeurIPS 2024 · 208 citations
  3. Long-Short Transformer: Efficient Transformers for Language and VisionChen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi et al.NeurIPS 2021 · 180 citations
  4. Exploring the Limits of Large Scale Pre-trainingSamira Abnar, Mostafa Dehghani, Behnam Neyshabur, Hanie SedghiICLR 2022 · 135 citations
  5. Continuous Diffusion Model for Language ModelingJaehyeong Jo, Sung Ju HwangNeurIPS 2025 · 30 citations

Builds on7

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  4. Generative Pretraining From PixelsMark Chen, Alec Radford, Rewon Child, Jeffrey Wu et al.ICML 2020 · 1,773 citations
  5. Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations

Related papers

  1. BEiT: BERT Pre-Training of Image TransformersHangbo Bao, Li Dong, Songhao Piao, Furu WeiICLR 2022 · 3,632 citations
  2. Adaptive Attention for Sparse-based Long-sequence TransformerXuanyu Zhang, Zhepeng Lv, Qing YangACL Findings 2023 · 6 citations
  3. Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token OrthogonalizationHuaibo Huang, Xiaoqiang Zhou, Ran HeNeurIPS 2022 · 35 citations
  4. CAB: Comprehensive Attention Benchmarking on Long Sequence ModelingJun Zhang, Shuyang Jiang, Jiangtao Feng, Lin Zheng et al.ICML 2023 · 11 citations
  5. Omni-Kernel Network for Image RestorationYuning Cui, Wenqi Ren, Alois KnollAAAI 2024 · 290 citations
Dusk over the sea between two cliffs drawn in fine vertical lines