Lune

EMNLP Findings2024

In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models

Ayrton San Joaquin, Bin Wang, Zhengyuan Liu, Philippe Muller, Nicholas Asher, Brian Lim, Nancy F. Chen

Abstract

Despite advancements, fine-tuning Large Language Models (LLMs) remains costly due to the extensive parameter count and substantial data requirements for model generalization. Accessibility to computing resources remains a barrier for the open-source community. To address this challenge, we propose the In2Core algorithm, which selects a coreset by analyzing the correlation between training and evaluation samples with a trained model. Notably, we assess the model's internal gradients to estimate this relationship, aiming to rank the contribution of each training point. To enhance efficiency, we propose an optimization to compute influence functions with a reduced number of layers while achieving similar accuracy. By applying our algorithm to instruction finetuning data of LLMs, we can achieve similar performance with just 50% of the training data. Meantime, using influence functions to analyze model coverage to certain testing samples could provide a reliable and interpretable signal on the training set's coverage of those test points.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection cites LoRA: Low-Rank Adaptation of Large Language Models
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
  • TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. IF-Guide: Influence Function-Guided Detoxification of LLMsZachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun HongNeurIPS 2025 · 9 citations
  2. Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data SelectionYang Zhao, Li Du, Xiao Ding, Yangou Ouyang et al.ACL 2025 · 3 citations
  3. TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction TuningManish Nagaraj, Sakshi Choudhary, Utkarsh Saxena, Deepak Ravikumar et al.ICML 2026 · 2 citations
  4. RRInf: Efficient Influence Function Estimation via Ridge Regression for Large Language Models and Text-to-Image Diffusion ModelsZhuozhuo Tu, Cheng Chen, Yuxuan DuEMNLP 2025

Builds on19

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations

Related papers

  1. Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning DataIshika Agarwal, Dilek Hakkani-TurNeurIPS 2025 · 2 citations
  2. BRIEF: Bi-level Coreset Selection for Efficient Instruction Tuning in LLMsChaoyuan Shen, Chi Zhang, Chengliang Chai, Jiacheng Wang et al.VLDB 2026 · 2 citations
  3. STAFF: Speculative Coreset Selection for Task-Specific Fine-tuningXiaoyu Zhang, Juan Zhai, Shiqing Ma, Chao Shen et al.ICLR 2025
  4. Quadratic Coreset Selection: Certifying and Reconciling Sequence and Token Mining for Efficient Instruction TuningZiliang Chen, Yongsen Zheng, Zhao-Rong Lai, Zhanfu Yang et al.NeurIPS 2025 · 4 citations
  5. Efficient Data Selection at Scale via Influence DistillationMahdi Nikdan, Vincent Cohen-Addad, Dan Alistarh, Vahab MirrokniNeurIPS 2025 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines