ACL Findings2022
Finding the Dominant Winning Ticket in Pre-Trained Language Models
Zhuocheng Gong, Di He, Yelong Shen, Tie-Yan Liu, Weizhu Chen, Dongyan Zhao, Ji-Rong Wen, Rui Yan
Abstract
The Lottery Ticket Hypothesis suggests that for any over-parameterized model, a small subnetwork exists to achieve competitive performance compared to the backbone architecture. In this paper, we study whether there is a winning lottery ticket for pre-trained language models, which allow the practitioners to fine-tune the parameters in the ticket but achieve good downstream performance. To achieve this, we regularize the fine-tuning process with L1 distance and explore the subnetwork structure (what we refer to as the "dominant winning ticket"). Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix. Strikingly, we find that a dominant winning ticket that takes up 0.05% of the parameters can already achieve satisfactory performance, indicating that the PLM is significantly reducible during fine-tuning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers cites The Lottery Ticket Hypothesis for Pre-trained BERT Networks
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers cites When BERT Plays the Lottery, All Tickets Are Winning
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models cites The Lottery Ticket Hypothesis for Pre-trained BERT Networks
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models cites When BERT Plays the Lottery, All Tickets Are Winning
- Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning cites The Lottery Ticket Hypothesis for Pre-trained BERT Networks
- Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning cites When BERT Plays the Lottery, All Tickets Are Winning
- The Lottery Ticket Hypothesis for Pre-trained BERT Networks cites Picking Winning Tickets Before Training by Preserving Gradient Flow
- The Lottery Ticket Hypothesis for Pre-trained BERT Networks cites When BERT Plays the Lottery, All Tickets Are Winning
- On the Power of Context-Enhanced Learning in LLMs cites Task-Specific Skill Localization in Fine-tuned Language Models
- Task-Specific Skill Localization in Fine-tuned Language Models cites The Lottery Ticket Hypothesis for Pre-trained BERT Networks
- Task-Specific Skill Localization in Fine-tuned Language Models cites When BERT Plays the Lottery, All Tickets Are Winning
Cited by top-tier papers4
- Task-Specific Skill Localization in Fine-tuned Language ModelsAbhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, Sanjeev AroraICML 2023 · 100 citations
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language ModelsZhong Zhang, Bang Liu, Junming ShaoACL 2023 · 8 citations
- Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt TuningZujie Liang, Feng Wei, Yin Jie, Yuxi Qian et al.ACL 2023 · 8 citations
- On the Power of Context-Enhanced Learning in LLMsXingyu Zhu, Abhishek Panigrahi, Sanjeev AroraICML 2025
Builds on9
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Picking Winning Tickets Before Training by Preserving Gradient FlowChaoqi Wang, Guodong Zhang, Roger B. GrosseICLR 2020 · 743 citations
- Compacter: Efficient Low-Rank Hypercomplex Adapter LayersRabeeh Karimi Mahabadi, James Henderson, Sebastian RuderNeurIPS 2021 · 700 citations
- The Lottery Ticket Hypothesis for Pre-trained BERT NetworksTianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu et al.NeurIPS 2020 · 428 citations
- When BERT Plays the Lottery, All Tickets Are WinningSai Prasanna, Anna Rogers, Anna RumshiskyEMNLP 2020 · 114 citations
Related papers
- Playing Lottery Tickets with Vision and LanguageZhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen et al.AAAI 2022 · 64 citations
- Super Tickets in Pre-Trained Language Models: From Model Compression to Improving GeneralizationChen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang et al.ACL 2021
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPHaonan Yu, Sergey Edunov, Yuandong Tian, Ari S. MorcosICLR 2020 · 156 citations
- Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?Ning Liu, Geng Yuan, Zhengping Che, Xuan Shen et al.ICML 2021 · 34 citations
- Dual Lottery Ticket HypothesisYue Bai, Huan Wang, Zhiqiang Tao, Kunpeng Li et al.ICLR 2022 · 49 citations
