SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training

Information-based data selection for instruction tuning is compelling: maximizing the log-determinant of the Fisher information yields a monotone submodular objective, enabling greedy algorithms to achieve a (1 -1/e) approximation under a cardinality budget. In practice, however, we identify alleviating gradient conflicts, misalignment between per-sample gradients, is a key factor that slows down the decay of marginal log-determinant information gains, thereby preventing significant loss of information. We formalize this via an ε-decomposition that quantifies the deviation from ideal submodularity as a function of conflict statistics, yielding data-dependent approximation factors that tighten as conflicts diminish. Guided by this analysis, we propose SPICE, a conflict-aware selector that maximizes information while penalizing misalignment, and that supports early stopping and proxy models for efficiency. Empirically, SPICE selects subsets with higher logdeterminant information than original criteria, and these informational gains translate into performance improvements: across 8 benchmarks with LLaMA2-7B and Qwen2-7B, SPICE uses only 10% of the data, yet matches or exceeds 6 methods including full-data tuning. This achieves performance improvements with substantially lower training cost. Code is available at code. Slower decay ⇒ larger total info Lower conflict ⇒ larger total info Figure 1: (a) Concept. At step t, the marginal information gain ∆ t is the incremental increase of Fisher Information utility when adding one sample under the current set S. Slower decay of ∆ t yields larger cumulative information under the same budget k. (b) Empirical. A low-conflict selection (conflict measured by negative cosine alignment to the mean gradient) exhibits slower decay and thus higher information utility at equal budgets.