Scaling Long Context Training Data by Long-Distance Referrals
Yonghao Zhuang, Lanxiang Hu, Longfei Yun, Souvik Kundu, Zhengzhong Liu, Eric P. Xing, Hao Zhang
Abstract
Training large language models for long context understanding faces the challenge of data shortage. Previous data engineering approaches mechanically concatenate short documents, which may create many pseudo long documents but raise concerns about data quality. In this paper, we study the core attribute of high quality data for long context training, and provide a data pipeline, LongPack, to scale such data. We found that long distance referrals, which occur in natural long documents, are crucial for long-context training. However, simply concatenating short documents does not reliably generate these relations. We further show that the density of long-distance referrals, which is higher in longer documents, has a key role in training efficiency, making previous upsampling methods suboptimal. To enrich long documents, we propose LongPack, a data pipeline that constructs long documents by packing shorter ones based on referral relationships. Specifically, for web pages, which are the primary source for language model training, we found hyper-link a native signal for such a relation. By packing web pages through their hyper-link connection, we can create longer, high-quality documents. Our experiments demonstrate that LongPack is highly scalable, generating a corpus of long documents equivalent in size to an entire pretraining dataset using just 0.5% root documents. Furthermore, the constructed documents have a 'near-natural' quality as innate long documents for long context training, reaching a 32.7% higher score than previous state-of-the-art methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Data Engineering for Scaling Language Models to 128K Context cites YaRN: Efficient Context Window Extension of Large Language Models
- Data Engineering for Scaling Language Models to 128K Context cites LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Builds on10
- YaRN: Efficient Context Window Extension of Large Language ModelsBowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico ShippoleICLR 2024 · 508 citations
- Scaling Data-Constrained Language ModelsNiklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao et al.NeurIPS 2023 · 475 citations
- RepoBench: Benchmarking Repository-Level Code Auto-Completion SystemsTianyang Liu, Canwen Xu, Julian J. McAuleyICLR 2024 · 338 citations
- Data Engineering for Scaling Language Models to 128K ContextYao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue et al.ICML 2024 · 204 citations
- LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
Related papers
- Fewer Truncations Improve Language ModelingHantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar et al.ICML 2024 · 28 citations
- Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language ModelsLongze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng et al.ACL 2024 · 4 citations
- Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining DataHaoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu et al.ICLR 2026 · 6 citations
- Structured Packing in LLM Training Improves Long Context UtilizationKonrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Yu Zhao et al.AAAI 2025 · 17 citations
- Hierachical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLMYongqiang Yao, Jingru Tan, Kaihuan Liang, Feizhao Zhang et al.NeurIPS 2025
