ACL Findings2026
Large-Scale Diverse Synthesis for Mid-Training
Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai
Abstract
Mid-training has become critical for enhancing the knowledge and reasoning ability of large language models (LLMs), especially through the utilization of large-scale synthetic data.However, existing data synthesis methods often generate simplistic and homogeneous QA pairs, with limited scale and diversity.To address this, we propose BoostQA, a novel framework designed to synthesize large-scale, diverse, and high-quality QA data for midtraining.BoostQA introduces model probes during mid-training for the first time and implements STEM-focused multi-grade synthesis to boost data diversity as well as high-difficulty synthesis to alleviate difficulty degradation, followed by answer refinement to further improve quality.Extensive experiments by mid-training Llama-3 8B demonstrate that using only 20Btoken BoostQA data achieves a significant average improvement of 12.74% on MMLU and CMMLU over the pre-training baseline.After mid-training on 500B tokens, including 100Btoken BoostQA data, our model achieves SOTA average results across benchmarks among mainstream models of comparable size.BoostQA also demonstrates robust scalability, with performance consistently improving as model size, data volume, and initial FLOPs scale. 1 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 MMLU 73.6 65.5 70.7 74.1 72.0 76.2 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 CMMLU 75.0 53.1 75.0 82.8 59.1 80.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on19
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
- Efficient large-scale language model training on GPU clusters using megatron-LMDeepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley et al.SC 2021 · 576 citations
- Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text ModelsJianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma et al.ACL Findings 2022 · 358 citations
Related papers
- LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge PointsXuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng et al.ACL 2026 · 3 citations
- Training with "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-Context TasksYijiong Yu, Yongfeng Huang, Zhixiao Qi, Zhe ZhouAAAI 2025 · 5 citations
- Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop LearningZinan Tang, Xin Gao, Qizhi Pei, Zhuoshi Pan et al.EMNLP 2025
- MASTER: Enhancing Large Language Model via Multi-Agent Simulated TeachingLiang Yue, Yihong Tang, Kehai Chen, Jie Liu et al.NeurIPS 2025 · 1 citation
- Oasis: One Image is All You Need for Multimodal Instruction Data SynthesisLetian Zhang, Quan Cui, Bingchen Zhao, Cheng YangICCV 2025
