Evaluating Very Long-Term Conversational Memory of LLM Agents
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang
Abstract
Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in longcontext large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very longterm dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LOCOMO, a dataset of very long-term conversations, each encompassing approx. 600 turns and 16K tokens on avg., over up to 32 sessions. Based on LOCOMO, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
99 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Efficient Streaming Language Models with Attention Sinks cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory cites Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory cites A-Mem: Agentic Memory for LLM Agents
- Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning cites A-Mem: Agentic Memory for LLM Agents
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions cites A-Mem: Agentic Memory for LLM Agents
- LightMem: Lightweight and Efficient Memory-Augmented Generation cites Generative Agents: Interactive Simulacra of Human Behavior
- LightMem: Lightweight and Efficient Memory-Augmented Generation cites Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
- LightMem: Lightweight and Efficient Memory-Augmented Generation cites A-Mem: Agentic Memory for LLM Agents
Cited by top-tier papers99
- A-Mem: Agentic Memory for LLM AgentsWujiang Xu, Zujie Liang, Kai Mei, Hang Gao et al.NeurIPS 2025 · 1,138 citations
- Evaluating Memory in LLM Agents via Incremental Multi-Turn InteractionsYuanzhe Hu, Yu Wang, Julian McAuleyICLR 2026 · 246 citations
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning MemorySiru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen et al.ICLR 2026 · 244 citations
- LightMem: Lightweight and Efficient Memory-Augmented GenerationJizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang et al.ICLR 2026 · 162 citations
- Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement LearningSikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie et al.ACL 2026 · 140 citations
Builds on31
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Retrieval Augmentation Reduces Hallucination in ConversationKurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela et al.EMNLP Findings 2021 · 579 citations
Related papers
- Evaluating the Long-Term Memory of Large Language ModelsZixi Jia, Qinghua Liu, Hexiao Li, Yuyan Chen et al.ACL Findings 2025 · 5 citations
- HyperMem: Hypergraph Memory for Long-Term ConversationsJuwei Yue, Chuanrui Hu, Jiawei Sheng, Zuyi Zhou et al.ACL 2026 · 4 citations
- Beyond Goldfish Memory: Long-Term Open-Domain ConversationJing Xu, Arthur Szlam, Jason WestonACL 2022 · 329 citations
- ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional SupportTiantian Chen, Jiaqi Lu, Ying Shen, Lin ZhangWWW 2026 · 1 citation
- Beyond Dialogue Time: Temporal Semantic Memory for Personalized LLM AgentsMiao Su, Yucan Guo, Zhongni Hou, Long Bai et al.ACL Findings 2026
