EMNLP Findings2020
Query-Key Normalization for Transformers
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, Yuxuan Chen
Abstract
Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply ℓ 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
99 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers99
- Unified Training of Universal Time Series Forecasting TransformersGerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong et al.ICML 2024 · 513 citations
- Genie: Generative Interactive EnvironmentsJake Bruce, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder et al.ICML 2024 · 513 citations
- TransNeXt: Robust Foveal Visual Perception for Vision TransformersDai ShiCVPR 2024 · 313 citations
- Memorizing TransformersYuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, Christian SzegedyICLR 2022 · 231 citations
- Focused Transformer: Contrastive Training for Context ScalingSzymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu et al.NeurIPS 2023 · 190 citations
Builds on3
- PowerNorm: Rethinking Batch Normalization in TransformersSheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney et al.ICML 2020 · 88 citations
- BPE-Dropout: Simple and Effective Subword RegularizationIvan Provilkov, Dmitrii Emelianenko, Elena VoitaACL 2020 · 17 citations
- Language Model Prior for Low-Resource Neural Machine TranslationChristos Baziotis, Barry Haddow, Alexandra BirchEMNLP 2020 · 11 citations
Related papers
- Binarized Neural Machine TranslationYichi Zhang, Ankush Garg, Yuan Cao, Lukasz Lew et al.NeurIPS 2023 · 21 citations
- Affine-Scaled Attention: Towards Flexible and Stable Transformer AttentionJeongin Bae, baeseong park, Gunho Park, Minsub Kim et al.ICML 2026 · 1 citation
- The Devil in Linear TransformerZhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li et al.EMNLP 2022 · 24 citations
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine TranslationAlessandro Raganato, Yves Scherrer, Jörg TiedemannEMNLP Findings 2020 · 88 citations
- Accelerating Neural Machine Translation with Partial Word Embedding CompressionFan Zhang, Mei Tu, Jinyao YanAAAI 2021 · 3 citations
