Data Swarms: Optimizable Generation of Synthetic Evaluation Data

We propose DATA SWARMS, an algorithm to optimize the generation of synthetic evaluation data and advance quantitative desiderata of LLM evaluation. We first train a swarm of initial data generators using existing data, and define various evaluation objectives to reflect the desired properties of evaluation (e.g., generate more difficult problems for the evaluated models) and quantitatively evaluate data generators. We then employ particle swarm optimization to optimize the swarm of data generators, where they collaboratively search through the model parameter space to find new generators that advance these objectives. We further extend it to ADVERSARIAL SWARMS, where the data generator swarm generates harder data while the test taker model swarm learns from such data, co-evolving dynamically for better data and models simultaneously. Extensive experiments demonstrate that DATA SWARMS outperforms eight data generation baselines across five evaluation objectives, while ADVERSARIAL SWARMS produce more robust learning of synthetic data and stronger generalization. Further analysis reveals that DATA SWARMS successfully optimizes compositions of multiple evaluation objectives and generalizes to new off-the-shelf LLMs, unseen at optimization time. Response from model 4: Here is how to solve the problem: 1. Analyze the filling rates: * Anna fills the cup at 2 ounces per minute for the first 20 minutes of filling. * Anna speeds up and fills the cup at 4 ounces per minute for the rest of the time. 2. Calculate the amount of liquid filled at each rate: * First 20 minutes: 20 minutes * 2 ounces/minute = 40 ounces * Remaining time: 60 minutes (total) -20 minutes = 40 minutes * Liquid filled during the last 40 minutes: 40 minutes * 4 ounces/minute = 160 ounces 3. Figure out how many ounces were filled in total: * 40 ounces + 160 ounces = 200 ounces filled total 4. Determine how long the cup overflows if it starts at 0 ounces: * The cup overflows by 200 ounces -25 ounces = 175 ounces * At an overflow rate of 0.5 ounces per minute, it would take 175 ounces / 0.5 ounces/minute = 350 minutes for the cup to overflow. 5. Determine the fraction of time the cup stayed full: * Time the cup is full to fill to 25 ounces: it takes 25 ounces / 2 ounces/minute = 12.5 minutes * The cup was full for 12.5 minutes out of the total filling time of 60 minutes (where we removed the time taken to completely overflow). * The fraction of time the cup stood full is 12.5 minutes / 60 minutes = 0.2083. 6. Convert to a percentage * 0.2083 * 100% = 20.83% Answer: The cup stayed full approximately 20.83% of the time. ✗ Response from model 4: Here's how to solve this problem: 1. Time Taken by Mr. James * Mr. James starts first. Let the time (in hours) when Mr. Mark catches Mr. James be t. * The distance travelled by Mr. James is 30t meters. 2. Time Taken by Mr. Mark * Mr. Mark takes t hours to reach where Mr. James was already at that moment. * The distance travelled by Mr. Mark is 25t meters. 3. Equate Distances * Since Mr. Mark catches up to Mr. James when they both are at the same point, their distances need to be equal: 30t = 25t 4. Solve for Time (t) * 5*t = 0 * t = 0 This means Mr. Mark will always lag behind Mr. James as he has a slower speed. Answer: Since Mr. James is faster, Mr. Mark cannot catch Mr. James in a long-distance swimming contest. ✓