EMNLP Findings2023
NarrativeXL: a Large-scale Dataset for Long-Term Memory Models
Abstract
We propose a new large-scale (nearly a million questions) ultra-long-context (more than 50,000 words average document length) reading comprehension dataset. Using GPT 3.5, we summarized each scene in 1,500 hand-curated fiction books from Project Gutenberg, which resulted in approximately 150 scene-level summaries per book. After that, we created a number of reading comprehension questions based on these summaries, including three types of multiple-choice scene recognition questions, as well as free-form narrative reconstruction questions. With 990,595 total questions, our dataset is an order of magnitude larger than the closest alternatives. Crucially, most questions have a known "retention demand", indicating how long-term of a memory is needed to answer them, which should aid long-term memory performance evaluation. We validate our data in four small-scale experiments: one with human labelers, and three with existing language models. We show that our questions 1) adequately represent the source material 2) can be used to diagnose a model's memory capacity 3) are not trivial for modern language models even when the memory demand does not exceed those models' context lengths. Lastly, we provide our code which can be used to further expand the dataset with minimal human labor.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA cites BERTScore: Evaluating Text Generation with BERT
- LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA cites NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens cites Long Range Arena : A Benchmark for Efficient Transformers
Cited by top-tier papers2
Builds on10
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations
- Compressive Transformers for Long-Range Sequence ModellingJack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier et al.ICLR 2020 · 833 citations
- Exploring Length Generalization in Large Language ModelsCem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz et al.NeurIPS 2022 · 267 citations
Related papers
- A Human-Inspired Reading Agent with Gist Memory of Very Long ContextsKuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John F. Canny et al.ICML 2024 · 106 citations
- BookWorm: A Dataset for Character Description and AnalysisArgyrios Papoudakis, Mirella Lapata, Frank KellerEMNLP Findings 2024 · 1 citation
- One Thousand and One Pairs: A "novel" challenge for long-context language modelsMarzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal et al.EMNLP 2024 · 6 citations
- Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMsMohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla et al.ICLR 2026 · 56 citations
- LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
