DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael K. Ng, Xin Jiang, Zhenguo Li, Yu Li
Abstract
Positional encoding plays a crucial role in transformers, significantly impacting model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of input data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and can be dynamically adjusted with the given attention. In this paper, we propose a Data-Adaptive Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances in terms of trained length and length generalization, where the improvements are statistically significant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
17 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Efficient Streaming Language Models with Attention Sinks cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Efficient Streaming Language Models with Attention Sinks cites YaRN: Efficient Context Window Extension of Large Language Models
- Efficient Streaming Language Models with Attention Sinks cites Language Models are Few-Shot Learners
- PaTH Attention: Position Encoding via Accumulating Householder Transformations cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- PaTH Attention: Position Encoding via Accumulating Householder Transformations cites YaRN: Efficient Context Window Extension of Large Language Models
- PaTH Attention: Position Encoding via Accumulating Householder Transformations cites DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
- Group Representational Position Encoding cites PaTH Attention: Position Encoding via Accumulating Householder Transformations
- Group Representational Position Encoding cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Group Representational Position Encoding cites YaRN: Efficient Context Window Extension of Large Language Models
- Mixture-of-Scores: Robust Image-Text Data Valuation via Three Lines of Code cites DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation cites Language Models are Few-Shot Learners
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation cites Efficient Streaming Language Models with Attention Sinks
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation cites YaRN: Efficient Context Window Extension of Large Language Models
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation cites Language Models are Few-Shot Learners
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites YaRN: Efficient Context Window Extension of Large Language Models
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding cites Language Models are Few-Shot Learners
Cited by top-tier papers17
- PaTH Attention: Position Encoding via Accumulating Householder TransformationsSonglin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan et al.NeurIPS 2025 · 36 citations
- Extending the Context of Pretrained LLMs by Dropping Their Positional EmbeddingYoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo CetinICLR 2026 · 13 citations
- DAPE V2: Process Attention Score as Feature Map for Length ExtrapolationChuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong et al.ACL 2025 · 12 citations
- Group Representational Position EncodingYifan Zhang, Zixiang Chen, Yifeng Liu, Zhen Qin et al.ICLR 2026 · 5 citations
- Mixture-of-Scores: Robust Image-Text Data Valuation via Three Lines of CodeSitong Wu, Haoru Tan, Yukang Chen, Shaofeng Zhang et al.ICCV 2025 · 4 citations
Builds on43
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
- YaRN: Efficient Context Window Extension of Large Language ModelsBowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico ShippoleICLR 2024 · 508 citations
Related papers
- Context-aware Biases for Length ExtrapolationAli Veisi, Hamidreza Amirzadeh, Amir MansourianEMNLP 2025 · 2 citations
- The Impact of Positional Encoding on Length Generalization in TransformersAmirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das et al.NeurIPS 2023 · 444 citations
- Conditional Positional Encodings for Vision TransformersXiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang et al.ICLR 2023 · 406 citations
- Mitigating Tokenization-Induced Distance Distortion in Long-Context Multilingual Machine TranslationKhotso Selialia, Antoine Nzeyimana, Fatima M. AnwarACL 2026
- LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without TrainingSikui Zhang, Guangze Gao, Ziyun Gan, Chunfeng Yuan et al.ACL Findings 2026
