Lune

ACL Findings2022

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Abstract

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/ thunlp/MoEfication.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

60 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Big Bird: Transformers for Longer Sequences cites Reformer: The Efficient Transformer
  • ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models cites Language Models are Few-Shot Learners
  • Locating and Editing Factual Associations in GPT cites Language Models are Few-Shot Learners
  • PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU cites ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
  • PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU cites Language Models are Few-Shot Learners
  • Blockwise Parallel Transformers for Large Context Models cites Reformer: The Efficient Transformer
  • Blockwise Parallel Transformers for Large Context Models cites Language Models are Few-Shot Learners
  • Blockwise Parallel Transformers for Large Context Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding cites Language Models are Few-Shot Learners
  • A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity cites Locating and Editing Factual Associations in GPT
  • A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers60

  1. A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and ToxicityAndrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg et al.ICML 2024 · 177 citations
  2. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language ModelsIman Mirzadeh, Keivan Alizadeh-Vahid, Sachin Mehta, Carlo C. del Mundo et al.ICLR 2024 · 109 citations
  3. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUYixin Song, Zeyu Mi, Haotong Xie, Haibo ChenSOSP 2024 · 86 citations
  4. Blockwise Parallel Transformers for Large Context ModelsHao Liu, Pieter AbbeelNeurIPS 2023 · 55 citations
  5. Towards the Law of Capacity Gap in Distilling Language ModelsChen Zhang, Qiuchi Li, Dawei Song, Zheyu Ye et al.ACL 2025 · 39 citations

Builds on17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  3. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  4. Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
  5. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations

Related papers

  1. Exploring the Benefit of Activation Sparsity in Pre-trainingZhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin et al.ICML 2024 · 6 citations
  2. Analytical FFN-to-MoE Restructuring via Activation Pattern AnalysisZehua Pei, Hui-Ling Zhen, Lancheng Zou, Xianzhi Yu et al.ACL 2026 · 6 citations
  3. DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMsMinxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong et al.EMNLP 2025
  4. Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts ConversionFilip Szatkowski, Bartosz Wójcik, Mikolaj Piórczynski, Simone ScardapaneNeurIPS 2024 · 19 citations
  5. Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language ModelZeyu Liu, Tim Dettmers, Xi Lin, Veselin Stoyanov et al.EMNLP 2023 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines