Wavelet-based Positional Representation for Long Context
Yui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito
Abstract
In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. This is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences. We analyzed conventional position encoding methods for long contexts and found the following characteristics. (1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. However, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). This limitation could explain why RoPE performs poorly in extrapolation. (2) Previous research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes. However, it has limitations in capturing deep dependencies because it restricts the receptive field of the model. From these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field. Experimental results show that this new method improves the performance of the model in both short and long contexts. In particular, our method allows extrapolation of position information without limiting the model's attention field.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation cites Language Models are Few-Shot Learners
- YaRN: Efficient Context Window Extension of Large Language Models cites Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation-Extrapolation Trade-off cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation-Extrapolation Trade-off cites YaRN: Efficient Context Window Extension of Large Language Models
- Probing Rotary Position Embeddings through Frequency Entropy cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites YaRN: Efficient Context Window Extension of Large Language Models
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites Language Models are Few-Shot Learners
- RePo: Language Models with Context Re-Positioning cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- RePo: Language Models with Context Re-Positioning cites Probing Rotary Position Embeddings through Frequency Entropy
- RePo: Language Models with Context Re-Positioning cites Language Models are Few-Shot Learners
Cited by top-tier papers4
- Extending the Context of Pretrained LLMs by Dropping Their Positional EmbeddingYoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo CetinICLR 2026 · 13 citations
- RePo: Language Models with Context Re-PositioningHuayang Li, Tianyu Zhao, Deng Cai, Richard SproatICML 2026 · 3 citations
- Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation-Extrapolation Trade-offYui Oka, Itsumi Saito, Kyosuke Nishida, Kuniko SaitoICLR 2026
- Probing Rotary Position Embeddings through Frequency EntropyYui Oka, Kentaro Hanafusa, Taku Hasegawa, Kyosuke Nishida et al.ICLR 2026
Builds on15
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional DomainsMatthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil et al.NeurIPS 2020 · 4,036 citations
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
- Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations
- YaRN: Efficient Context Window Extension of Large Language ModelsBowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico ShippoleICLR 2024 · 508 citations
Related papers
- On the token distance modeling ability of higher RoPE attention dimensionXiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng et al.EMNLP Findings 2024 · 2 citations
- Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length GeneralizationErmo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang et al.ICML 2025
- Beyond Position: the emergence of wavelet-like properties in TransformersValeria Ruscio, Umberto Nanni, Fabrizio SilvestriACL 2025 · 1 citation
- LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without TrainingSikui Zhang, Guangze Gao, Ziyun Gan, Chunfeng Yuan et al.ACL Findings 2026
- Scaling Laws of RoPE-based ExtrapolationXiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu et al.ICLR 2024 · 130 citations
