OPIC: Enhancing Language Model Merging via Optimizing In-Context Capability

Task-vector-based model merging enables lowcost, training-free multi-task learning for large language models, but suffers from performance degradation compared to individually fine-tuned models. Prior mitigation strategies largely rely on validation data for costly hyperparameter tuning, limiting both interpretability and practicality. We therefore propose OPIC, an evolutionary optimization-based model merging framework. Our preliminary experiments reveal that the degradation of in-context learning (ICL) capabilities exhibits a strong correlation with performance deterioration. Motivated by this insight, we formulate model merging as an optimization problem with ICL preservation as the objective. OPIC introduces a hierarchical refinement operators and optimizes it using self-generated data, effectively eliminating the reliance on external validation sets. Experimental results demonstrate that OPIC achieves an average performance retention of 80.73%, outperforming SOTA methods and improving by up to 11.1% over recent validation-free approaches. In addition, OPIC is compatible with existing merging pipelines, offering a new alternative solution for deploying without validation dependencies. Code is available at: https://github.com/illusion-hj/OPIC