Lune

ACL Findings2024

Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge

Daniel Mela, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas

Abstract

Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • Boosting Resilience of Large Language Models through Causality-Driven Robust Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Boosting Resilience of Large Language Models through Causality-Driven Robust Optimization cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • Boosting Resilience of Large Language Models through Causality-Driven Robust Optimization cites DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
  • Boosting Resilience of Large Language Models through Causality-Driven Robust Optimization cites Training language models to follow instructions with human feedback
  • DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models cites Improving Language Models by Retrieving from Trillions of Tokens
  • DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models cites Training language models to follow instructions with human feedback
  • The Logical Implication Steering Method for Conditional Interventions on Transformer Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Boosting Resilience of Large Language Models through Causality-Driven Robust OptimizationXiaoling Zhou, Mingjie Zhang, Zhemg Lee, Yuncheng Hua et al.NeurIPS 2025 · 5 citations
  2. Editing Across Languages: A Survey of Multilingual Knowledge EditingNadir Durrani, Basel Mousi, Fahim DalviEMNLP 2025
  3. The Logical Implication Steering Method for Conditional Interventions on Transformer GenerationDamjan KalajdzievskiICML 2025

Builds on24

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  4. Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
  5. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language ModelsYung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim et al.ICLR 2024 · 354 citations

Related papers

  1. EAMET: Robust Massive Model Editing via Embedding Alignment OptimizationYanbo Dai, Zhenlan Ji, Zongjie Li, Shuai WangICLR 2026
  2. Massive Editing for Large Language Models via Meta LearningChenmien Tan, Ge Zhang, Jie FuICLR 2024 · 68 citations
  3. InstructEd: Soft-Instruction Tuning for Model Editing with HopsXiaoqi Han, Ru Li, Xiaoli Li, Jiye Liang et al.ACL Findings 2024 · 1 citation
  4. PMET: Precise Model Editing in a TransformerXiaopeng Li, Shasha Li, Shezheng Song, Jing Yang et al.AAAI 2024 · 208 citations
  5. Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language ModelsZihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou et al.ACL 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines