LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. Long-Bench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including singledoc QA, multi-doc QA, summarization, fewshot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on Long-Bench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other opensourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability. itory level demand the ability to model long con-045 text sequences that span thousands or even tens of 046 thousands of tokens in length. However, many of 047 today's large language models can only compre-048 hend and generate texts a few thousand tokens long, 049 leaving room for potential improvements in pro-050 cessing longer contexts. More recently, there has 051 been an increasing effort to improve large language 052 models' capabilities on long context understanding.