EMNLP Findings2023
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
Abstract
In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient finetuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model. To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their importance scores. Our experiment results with three different pretrained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- BrainMoE: Cognition Joint Embedding via Mixture-of-Expert Towards Robust Brain Foundation ModelZiquan Wei, Tingting Dan, Tianlong Chen, Guorong WuNeurIPS 2025 · 2 citations
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family ExpertsGuorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen et al.ICLR 2025
Builds on20
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
- Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations
- Relational Graph Attention Network for Aspect-based Sentiment AnalysisKai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan et al.ACL 2020 · 614 citations
Related papers
- MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse UpcyclingRachel S. Y. Teo, Tan Minh NguyenICLR 2025
- AdaMix: Mixture-of-Adaptations for Parameter-efficient Model TuningYaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu et al.EMNLP 2022 · 65 citations
- MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language ModelsJie Cao, Tianwei Lin, Bo Yuan, Rolan Yan et al.ACL 2026 · 2 citations
- Composing Parameter-Efficient Modules with Arithmetic OperationJinghan Zhang, Shiqi Chen, Junteng Liu, Junxian HeNeurIPS 2023 · 164 citations
- Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language ModelsZihan Wang, Deli Chen, Damai Dai, Runxin Xu et al.EMNLP 2024 · 2 citations
