EMNLP Findings2025
Do Influence Functions Work on Large Language Models?
Abstract
Influence functions are important for quantifying the impact of individual training data points on a model's predictions.Although extensive research has been conducted on influence functions in traditional machine learning models, their application to large language models (LLMs) has been limited.In this work, we conduct a systematic study to address a key question: do influence functions work on LLMs?Specifically, we evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings.Our further investigation reveals that their poor performance can be attributed to: (1) inevitable approximation errors when estimating the iHVP component due to the scale of LLMs, (2) uncertain convergence during fine-tuning, and, more fundamentally, (3) the definition itself, as changes in model parameters do not necessarily correlate with changes in LLM behavior.Thus, our study suggests the need for alternative approaches for identifying influential samples. The code is available at https://github.com/plumprc/ Failures-of-Influence-Functions-in-LLMs
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs cites Training language models to follow instructions with human feedback
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler cites LoRA: Low-Rank Adaptation of Large Language Models
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation cites LoRA: Low-Rank Adaptation of Large Language Models
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation cites QLoRA: Efficient Finetuning of Quantized LLMs
- LIMA: Less Is More for Alignment cites Training language models to follow instructions with human feedback
- Fast Data Attribution for Text-to-Image Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
- IF-Guide: Influence Function-Guided Detoxification of LLMs cites Training language models to follow instructions with human feedback
Cited by top-tier papers6
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data SchedulerZixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei et al.NeurIPS 2025 · 16 citations
- Better Training Data Attribution via Better Inverse Hessian-Vector ProductsAndrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae et al.NeurIPS 2025 · 12 citations
- IF-Guide: Influence Function-Guided Detoxification of LLMsZachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun HongNeurIPS 2025 · 9 citations
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence EstimationDmytro Vitel, Anshuman ChhabraICLR 2026 · 8 citations
- Fast Data Attribution for Text-to-Image ModelsSheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Richard Zhang et al.NeurIPS 2025 · 7 citations
Builds on18
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
Related papers
- Understanding Impact of Human Feedback via Influence FunctionsTaywon Min, Haeone Lee, Yongchan Kwon, Kimin LeeACL 2025 · 11 citations
- RRInf: Efficient Influence Function Estimation via Ridge Regression for Large Language Models and Text-to-Image Diffusion ModelsZhuozhuo Tu, Cheng Chen, Yuxuan DuEMNLP 2025
- In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language ModelsAyrton San Joaquin, Bin Wang, Zhengyuan Liu, Philippe Muller et al.EMNLP Findings 2024 · 3 citations
- Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning ModelsAnshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra et al.ICML 2025
- EULoInf: Efficient Hessian-Free Entropy Based Uncertainty-Aware Data Influence ApproximationRunxin Cai, Jingtan Wang, Bryan Kian Hsiang LowACL Findings 2026
