What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices

Recent advancements in large language models (LLMs) with extended context windows have significantly improved various tasks. To improve long-context capabilities, much work focuses on augmenting LLM's capabilities with synthetic data. Existing methods often leverage the Self-Instruct framework to generate longcontext instruction-tuning data. However, our preliminary experiments show that fewer than 35% of samples generated by Qwen-2 72B are multi-hop, and over 40% exhibit poor quality, limiting comprehensive understanding and further research. To address this, we propose the Multi-agent Interactive Multi-hop Generation (MIMG) framework, which integrates a quality verification agent, a single-hop question generation agent, a multiple question sampling strategy, and a multi-hop question merger agent. This framework significantly improves data quality, with high-quality, multi-hop, and diverse data. Furthermore, we conduct a thorough analysis of document selection, question merging, and validation techniques through extensive experiments across various models. Our results demonstrate that synthetic highquality long-context instruction data can enhance model performance, surpassing even models trained on larger amounts of humanannotated data. Our code and relevant data are available at: https://github.com/ WowCZ/LongMIT . Document 1 Question 11 Answer 11 … Question Semantic Relevance Matrix Multi-hop Question 1 Multi-hop Answer 1 Intra-Document Multi-hop Data Multi-hop Question 2 Multi-hop Answer 2 Inter-Document Multi-hop Data Doc 1 Doc 2 Doc 3 Q11 Q12 Q13 Q14 Q21 Q22 Q23 Q 31 Q 32 Doc 1 Doc 2 Doc 3 Q11 Q12 Q13 Q14