EMNLP Findings2023
RWKV: Reinventing RNNs for the Transformer Era
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella
Abstract
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
136 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- VMamba: Visual State Space Model cites MLP-Mixer: An all-MLP Architecture for Vision
- xLSTM: Extended Long Short-Term Memory cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites MLP-Mixer: An all-MLP Architecture for Vision
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites xLSTM: Extended Long Short-Term Memory
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites Big Bird: Transformers for Longer Sequences
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Cited by top-tier papers136
- VMamba: Visual State Space ModelYue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu et al.NeurIPS 2024 · 3,199 citations
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao, Albert GuICML 2024 · 1,407 citations
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence DiffusionBoyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz et al.NeurIPS 2024 · 751 citations
- xLSTM: Extended Long Short-Term MemoryMaximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer et al.NeurIPS 2024 · 703 citations
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse AttentionHuiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu et al.NeurIPS 2024 · 479 citations
Builds on21
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- MLP-Mixer: An all-MLP Architecture for VisionIlya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer et al.NeurIPS 2021 · 3,862 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
Related papers
- Cross-lingual Open-Retrieval Question Answering for African LanguagesOdunayo Ogundepo, Tajuddeen Gwadabe, Clara Rivera, Jonathan H. Clark et al.EMNLP Findings 2023 · 4 citations
- Can Language Models Follow Multiple Turns of Entangled Instructions?Chi Han, Xin Liu, Haodong Wang, Shiyang Li et al.EMNLP Findings 2025
- AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard NegativesYanxi Chen, Wenhui Zhu, Xiwen Chen, Zhipeng Wang et al.ACL Findings 2026
- MorphoBench: A Benchmark with Difficulty Adaptive to Model ReasoningXukai Wang, Xuanbo Liu, Mingrui Chen, Haitian Zhong et al.ACL Findings 2026
- Do Vision-Language Models Have Internal World Models? Towards an Atomic EvaluationQiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen et al.ACL Findings 2025 · 3 citations
