How to Synthesize Text Data without Model Collapse?

Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic data will fundamentally reshape the web data ecosystem. Future GPT-n models will inevitably be trained on a blend of synthetic and human-produced data. In this paper, we focus on two questions: what is the impact of synthetic data on language model training, and how to synthesize data without model collapse? We first pre-train language models across different proportions of synthetic data, revealing a negative correlation between the proportion of synthetic data and model performance. We further conduct statistical analysis on synthetic data to uncover distributional shift phenomenon and over-concentration of n-gram features. Inspired by the above findings, we propose token editing on human-produced data to obtain semi-synthetic data. As a proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conduct extensive experiments on pre-training from scratch, continual pre-training, and supervised fine-tuning. The results validate our theoretical proof that token-level editing improves model performance. How to Synthesize Text Data without Model Collapse? ① Model Collapse Setting ⇢ 𝑬 𝒕𝒆𝒔𝒕 = 𝝈 𝟐 𝒅 𝑻'𝒅'𝟏 × 𝒏 ② Token-Level Editing ⇢ 𝑬 𝒕𝒆𝒔𝒕 ≤ 𝝈 𝟐 𝒅 𝑻'𝒅'𝟏 × 𝟐 ⇢ Avoiding Model Collapse 𝐷𝑎𝑡𝑎 ! Training 𝑓 ) Synthesizing 𝐷𝑎𝑡𝑎 " … 𝑓 * 𝑓 + Training 𝑓 ) 𝐷𝑎𝑡𝑎 # 𝐷𝑎𝑡𝑎 ! Training 𝑓 + Training Editing 𝑀 * 𝐷𝑎𝑡𝑎 * + (1 -𝑀 * )𝐷𝑎𝑡𝑎 ) … Source Real Data: 𝐷𝑎𝑡𝑎 ! Test Error 𝐸 "#$" Editing Operation Matrix M % Iterations 𝑖 ∈ 1, … , 𝑛