ACL Findings2025
MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, Zhenhua Dong
Abstract
Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLMbased agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https: //github.com/import-myself/Membench.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MemoryBank: Enhancing Large Language Models with Long-Term Memory cites Long Time No See! Open-Domain Conversation with Long-Term Persona Memory
- ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models cites Evaluating Very Long-Term Conversational Memory of LLM Agents
- Evaluating Very Long-Term Conversational Memory of LLM Agents cites MemoryBank: Enhancing Large Language Models with Long-Term Memory
- Evaluating Very Long-Term Conversational Memory of LLM Agents cites Generative Agents: Interactive Simulacra of Human Behavior
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions cites MemoryBank: Enhancing Large Language Models with Long-Term Memory
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions cites Generative Agents: Interactive Simulacra of Human Behavior
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions cites Evaluating Very Long-Term Conversational Memory of LLM Agents
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions cites CloneMem: Benchmarking Long-Term Memory for AI Clones
- Lightweight LLM Agent Memory with Small Language Models cites MemoryBank: Enhancing Large Language Models with Long-Term Memory
- Lightweight LLM Agent Memory with Small Language Models cites Generative Agents: Interactive Simulacra of Human Behavior
- Lightweight LLM Agent Memory with Small Language Models cites Evaluating Very Long-Term Conversational Memory of LLM Agents
- Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information cites MemoryBank: Enhancing Large Language Models with Long-Term Memory
- CloneMem: Benchmarking Long-Term Memory for AI Clones cites Evaluating Very Long-Term Conversational Memory of LLM Agents
Cited by top-tier papers9
- Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized InformationZeyu Zhang, Yang Zhang, Haoran Tan, Rui Li et al.KDD 2026 · 11 citations
- KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital CompanionsTingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang et al.ACL 2026 · 6 citations
- CloneMem: Benchmarking Long-Term Memory for AI ClonesSen Hu, Zhiyu Zhang, Yuxiang Wei, Xueran Han et al.ACL 2026 · 6 citations
- Lightweight LLM Agent Memory with Small Language ModelsJiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang et al.ACL 2026 · 4 citations
- ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language ModelsChonghan Qin, Xiachong Feng, Weitao Ma, Xiaocheng Feng et al.ACL 2026 · 1 citation
Builds on8
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- MemoryBank: Enhancing Large Language Models with Long-Term MemoryWanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye et al.AAAI 2024 · 394 citations
- LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
- Long Time No See! Open-Domain Conversation with Long-Term Persona MemoryXinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu et al.ACL Findings 2022 · 65 citations
- Evaluating Very Long-Term Conversational Memory of LLM AgentsAdyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal et al.ACL 2024 · 30 citations
Related papers
- Evaluating Memory in LLM Agents via Incremental Multi-Turn InteractionsYuanzhe Hu, Yu Wang, Julian McAuleyICLR 2026 · 246 citations
- From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized AgentsMd Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral et al.ACL Findings 2026 · 2 citations
- Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile AgentsShihan Deng, Weikai Xu, Hongda Sun, Wei Liu et al.ACL 2024 · 10 citations
- Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM AgentsYuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao et al.ACL 2026 · 23 citations
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic ApplicationsYujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan et al.ICML 2026 · 40 citations
