EMNLP Findings2025
GenPoE: Generative Passage-level Mixture of Experts for Knowledge Enhancement of LLMs
Xuebing Liu, Shanbao Qiao, Seung-Hoon Na
摘要
Typically, parametric adaptation methods such as domain-adaptive pretraining (DAP) and retrieval-augmented generation (RAG) have been considered effective approaches for adapting large language models (LLMs) to new knowledge or domains. To unify positive effects of parametric adaptation and RAG, this paper proposes GenPoE, i.e., "generative" passage-level mixture of experts (MoEs) for enhancing knowledge of LLMs. The key component is its novel MoE-generating hypernetwork which takes in-context retrieved passages and generates their "expert" parameters, where these generated parameters are then integrated into LLMs by forming expert networks. With its use of "generated" parameters, GenPoE does not require a separate parameter training or finetuning stage, which is often costly. By parameterizing passages into expert networks, Gen-PoE likely exhibits robustness even when the retrieved passages are irrelevant. Experiment results in two open-domain question answering (QA) tasks present that GenPoE shows improved performances over other passage-level knowledge editing, and its combination of RAG produces superior performances over RAG. Our data and code will be available at https: //github.com/Liu-Xuebing/GenPoE.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
- LoRA: Low-Rank Adaptation of Large Language Models 引用了 GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Fast Model Editing at Scale 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts 引用了 GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
它引用的 Paper27
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni 等NeurIPS 2020 · 19,162 次引用
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen 等ICLR 2021 · 1,954 次引用
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong 等ICML 2022 · 1,173 次引用
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn 等ICLR 2022 · 527 次引用
相关 Paper
- MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAGXuebing Liu, Shanbao Qiao, Roseline Nyange, Dongwook Min 等ICLR 2026
- QPaug: Question and Passage Augmentation for Open-Domain Question Answering of LLMsMinsang Kim, Cheoneum Park, Seung BaekEMNLP Findings 2024 · 5 次引用
- REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question AnsweringYuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao 等EMNLP 2024 · 12 次引用
- Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMsXin Zhou, Ping Nie, Yiwen Guo, Haojie Wei 等EMNLP 2024
- Parametric Retrieval Augmented GenerationWeihang Su, Yichen Tang, Qingyao Ai, Junxi Yan 等SIGIR 2025 · 25 次引用
