LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

Current long-context benchmarks primarily focus on retrieval-based tests, requiring Large Language Models (LLMs) to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark. Long-context generation refers to the ability of a language model to generate coherent and contextually accurate text that spans across lengthy passages or documents. While recent studies show strong performance on NIAH and other retrieval-based long-context benchmarks, there is a significant lack of benchmarks for evaluating long-context generation capabilities. To bridge this gap and offer a comprehensive assessment, we introduce a synthetic benchmark, LongGenBench, which is designed to evaluate the long-context generation capabilities of large language models (LLMs), with a particular focus on consistency in logical flow. LongGenBench redesigning the format of questions and necessitating that LLMs respond with a single, cohesive longcontext answer. Upon extensive evaluation using LongGenBench, we observe that: (1) both API accessed and open source models exhibit performance degradation in long-context generation scenarios, ranging from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of performance degradation, with the GEMINI-1.5-FLASH model showing the least degradation among API accessed models, and the QWEN2 series exhibiting the least degradation in LongGenBench among open source models.