EMNLP Findings2025
GenPoE: Generative Passage-level Mixture of Experts for Knowledge Enhancement of LLMs
Xuebing Liu, Shanbao Qiao, Seung-Hoon Na
Abstract
Typically, parametric adaptation methods such as domain-adaptive pretraining (DAP) and retrieval-augmented generation (RAG) have been considered effective approaches for adapting large language models (LLMs) to new knowledge or domains. To unify positive effects of parametric adaptation and RAG, this paper proposes GenPoE, i.e., "generative" passage-level mixture of experts (MoEs) for enhancing knowledge of LLMs. The key component is its novel MoE-generating hypernetwork which takes in-context retrieved passages and generates their "expert" parameters, where these generated parameters are then integrated into LLMs by forming expert networks. With its use of "generated" parameters, GenPoE does not require a separate parameter training or finetuning stage, which is often costly. By parameterizing passages into expert networks, Gen-PoE likely exhibits robustness even when the retrieved passages are irrelevant. Experiment results in two open-domain question answering (QA) tasks present that GenPoE shows improved performances over other passage-level knowledge editing, and its combination of RAG produces superior performances over RAG. Our data and code will be available at https: //github.com/Liu-Xuebing/GenPoE.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Fast Model Editing at Scale cites LoRA: Low-Rank Adaptation of Large Language Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Builds on27
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
Related papers
- MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAGXuebing Liu, Shanbao Qiao, Roseline Nyange, Dongwook Min et al.ICLR 2026
- QPaug: Question and Passage Augmentation for Open-Domain Question Answering of LLMsMinsang Kim, Cheoneum Park, Seung BaekEMNLP Findings 2024 · 5 citations
- REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question AnsweringYuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao et al.EMNLP 2024 · 12 citations
- Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMsXin Zhou, Ping Nie, Yiwen Guo, Haojie Wei et al.EMNLP 2024
- Parametric Retrieval Augmented GenerationWeihang Su, Yichen Tang, Qingyao Ai, Junxi Yan et al.SIGIR 2025 · 25 citations
