M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models

Managing long sequences has become an important and necessary feature for large language models (LLMs). However, assessing their ability to handle long contexts remains a challenge. This paper introduces M 4 LE, a Multi-ability, Multi-range, Multi-task, Multi-domain benchmark for Long-context Evaluation. It encompasses 36 NLP datasets, covering 11 types of tasks and 12 domains, providing a comprehensive test bed. To address the lack of tasks featuring naturally long sequences, we propose an automatic approach to convert short-sequence tasks into long-sequence scenarios. These scenarios evaluate LLMs' long-context understanding across five key abilities: understanding of single or multiple relevant spans in long contexts based on explicit or semantic hints, and global context understanding. This automatic approach allows us to create instances evenly distributed from 1k to 8k input length. 1 Our evaluation of 11 prominent LLMs reveals that 1) Current LLMs struggle to understand long context, particularly when tasks require multiple-span attention. 2) Semantic retrieval is more difficult for competent LLMs. 3) Models fine-tuned on longer text with position interpolation have comparable performance to those using Neural Tangent Kernel (NTK) aware scaling methods without fine-tuning. We make our benchmark publicly available to encourage future research in this challenging area 2 . CHAPTER I On a hill by the Mississippi … For months no male emerged from the mass. … CHAPTER II It was a frail and blue and lonely Carol who…. … CHAPTER X …said Kennicott, as he unpacked his suit-case. … Explicit Single-Span Semantic Single-Span Explicit Multi-Span Semantic Multi-Span Global Span CHAPTER I On a hill by the Mississippi … For months no male emerged from the mass. … CHAPTER II It was a frail and blue and lonely Carol who…. … CHAPTER X …said Kennicott, as he unpacked his suit-case. … Summarize CHAPTER I. CHAPTER I On a hill by the Mississippi … For months no male emerged from the mass. … CHAPTER II It was a frail and blue and lonely Carol who…. … CHAPTER X …said Kennicott, as he unpacked his suit-case. … Summarize the chapter about Carol. Summarize the first and the last chapters. Summarize the chapters about Carol as well as Kennicott. CHAPTER I On a hill by the Mississippi … For months no male emerged from the mass. … CHAPTER II It was a frail and blue and lonely Carol who…. … CHAPTER X …said Kennicott, as he unpacked his suit-case. … Summarize the whole article. CHAPTER I On a hill by the Mississippi … For months no male emerged from the mass. … CHAPTER II It was a frail and blue and lonely Carol who…. … CHAPTER X …said Kennicott, as he unpacked his suit-case. …