Lune

EMNLP Findings2024

Head-wise Shareable Attention for Large Language Models

Zouying Cao, Yifei Yang, Hai Zhao

Abstract

Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. However, current weight sharing techniques primarily focus on small-scale models like BERT and employ coarse-grained sharing rules, e.g., layerwise. This becomes limiting given the prevalence of LLMs and sharing an entire layer or block obviously diminishes the flexibility of weight sharing. In this paper, we present a perspective on head-wise shareable attention for large language models. We further propose two memory-efficient methods that share parameters across attention heads, with a specific focus on LLMs. Both of them use the same dynamic strategy to select the shared weight matrices. The first method directly reuses the pre-trained weights without retraining, denoted as Direct-Share. The second method first post-trains with constraint on weight matrix similarity and then shares, denoted as PostShare. Experimental results reveal our head-wise shared models still maintain satisfactory capabilities, demonstrating the feasibility of fine-grained weight sharing applied to LLMs 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • LESA: Learnable LLM Layer Scaling-Up cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • LESA: Learnable LLM Layer Scaling-Up cites Measuring Massive Multitask Language Understanding
  • LESA: Learnable LLM Layer Scaling-Up cites Language Models are Few-Shot Learners
  • Measuring Massive Multitask Language Understanding cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Measuring Massive Multitask Language Understanding cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • BiCycle: Group-wise Recursive Transformer Based on ASR Mechanism cites LoRA: Low-Rank Adaptation of Large Language Models
  • MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. LESA: Learnable LLM Layer Scaling-UpYifei Yang, Zouying Cao, Xinbei Ma, Yao Yao et al.ACL 2025 · 6 citations
  2. BiCycle: Group-wise Recursive Transformer Based on ASR MechanismMin Ho Jang, Eun Seo Seo, Jin Young Kim, Hyeongsoo Lim et al.AAAI 2026
  3. MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of ShardsSheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong et al.ICLR 2025

Builds on25

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  5. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations

Related papers

  1. Beyond Linear Approximations: A Novel Pruning Approach for Attention MatrixYingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song et al.ICLR 2025
  2. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use CasesZechun Liu, Changsheng Zhao, Forrest N. Iandola, Chen Lai et al.ICML 2024 · 227 citations
  3. Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model CompressionJingcun Wang, Yu-Guang Chen, Ing-Chao Lin, Bing Li et al.ICLR 2025
  4. Memory-Efficient KV Cache Optimization for Large Language Model Inference at the EdgeChi Zhang, Haisheng Tan, Haotian Pan, Yang Xu et al.INFOCOM 2026 · 1 citation
  5. APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language ModelsZiyi Guan, Hantao Huang, Yupeng Su, Hong Huang et al.DAC 2024 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines