LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the lowdata regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https: //github.com/SqueezeAILab/LLM2LLM . Step 1 Train on the dataset Training Data Finetune Step 3 Generate additional data with the wrong examples LLM2LLM Data Question: When the lengths of each side of a rectangle are 12 and 5, what is the length of the diagonal? Answer: sqrt(12 ** 2 + 5 ** 2) = 13 Question: What is the cumulative sum of all the integers starting from 1 to 5? Answer: 1 + 2 + … + 5 = (1 + 5) * 5 / 2 = 15 Evaluate + Student Model Seed Data Step 2 Evaluate on the dataset Question: A right triangle has two sides of length 3 and 4. What's the length of the hypothenuse? Answer: 3 + 4 = 7 Question: If Lisa has 5 apples and Tom gives her 8 more apples, how many apples does Lisa have in total? Answer: 5 + 8 = 13