Lune

ACL2023Top-tier venue

Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis

Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, Peter J. Ramadge

Abstract

Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create Sandwich, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

23 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ETC: Encoding Long and Structured Inputs in Transformers cites Big Bird: Transformers for Longer Sequences
  • Do Vision Transformers See Like Convolutional Neural Networks? cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • The Impact of Positional Encoding on Length Generalization in Transformers cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
  • Base of RoPE Bounds Context Length cites The Impact of Positional Encoding on Length Generalization in Transformers
  • Training-Free Long-Context Scaling of Large Language Models cites The Impact of Positional Encoding on Length Generalization in Transformers
  • Training-Free Long-Context Scaling of Large Language Models cites Functional Interpolation for Relative Positions improves Long Context Transformers
  • Functional Interpolation for Relative Positions improves Long Context Transformers cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
  • Functional Interpolation for Relative Positions improves Long Context Transformers cites The Impact of Positional Encoding on Length Generalization in Transformers
  • Functional Interpolation for Relative Positions improves Long Context Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Transformers Can Do Arithmetic with the Right Embeddings cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
  • Transformers Can Do Arithmetic with the Right Embeddings cites The Impact of Positional Encoding on Length Generalization in Transformers
  • Transformers Can Do Arithmetic with the Right Embeddings cites Functional Interpolation for Relative Positions improves Long Context Transformers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers23

  1. The Impact of Positional Encoding on Length Generalization in TransformersAmirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das et al.NeurIPS 2023 · 444 citations
  2. Transformers Can Do Arithmetic with the Right EmbeddingsSean McLeish, Arpit Bansal, Alex Stein, Neel Jain et al.NeurIPS 2024 · 94 citations
  3. Training-Free Long-Context Scaling of Large Language ModelsChenxin An, Fei Huang, Jun Zhang, Shansan Gong et al.ICML 2024 · 68 citations
  4. Functional Interpolation for Relative Positions improves Long Context TransformersShanda Li, Chong You, Guru Guruganesh, Joshua Ainslie et al.ICLR 2024 · 66 citations
  5. Base of RoPE Bounds Context LengthMingyu Xu, Xin Men, Bingning Wang, Qingyu Zhang et al.NeurIPS 2024 · 56 citations

Builds on7

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  3. Do Vision Transformers See Like Convolutional Neural Networks?Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang et al.NeurIPS 2021 · 1,553 citations
  4. Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
  5. ETC: Encoding Long and Structured Inputs in TransformersJoshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek et al.EMNLP 2020 · 268 citations

Related papers

  1. Exploring Transformer ExtrapolationZhen Qin, Yiran Zhong, Hui DengAAAI 2024 · 12 citations
  2. Wavelet-based Positional Representation for Long ContextYui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko SaitoICLR 2025
  3. Length Extrapolation of Transformers: A Survey from the Perspective of Positional EncodingLiang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong et al.EMNLP Findings 2024 · 14 citations
  4. A Length-Extrapolatable TransformerYutao Sun, Li Dong, Barun Patra, Shuming Ma et al.ACL 2023 · 45 citations
  5. KERPLE: Kernelized Relative Positional Embedding for Length ExtrapolationTa-Chung Chi, Ting-Han Fan, Peter J. Ramadge, Alexander RudnickyNeurIPS 2022 · 112 citations
Dusk over the sea between two cliffs drawn in fine vertical lines