EMNLP Findings2025
Do Influence Functions Work on Large Language Models?
摘要
Influence functions are important for quantifying the impact of individual training data points on a model's predictions.Although extensive research has been conducted on influence functions in traditional machine learning models, their application to large language models (LLMs) has been limited.In this work, we conduct a systematic study to address a key question: do influence functions work on LLMs?Specifically, we evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings.Our further investigation reveals that their poor performance can be attributed to: (1) inevitable approximation errors when estimating the iHVP component due to the scale of LLMs, (2) uncertain convergence during fine-tuning, and, more fundamentally, (3) the definition itself, as changes in model parameters do not necessarily correlate with changes in LLM behavior.Thus, our study suggests the need for alternative approaches for identifying influential samples. The code is available at https://github.com/plumprc/ Failures-of-Influence-Functions-in-LLMs
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 6 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- LoRA: Low-Rank Adaptation of Large Language Models 引用了 Language Models are Few-Shot Learners
- QLoRA: Efficient Finetuning of Quantized LLMs 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs 引用了 Training language models to follow instructions with human feedback
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation 引用了 QLoRA: Efficient Finetuning of Quantized LLMs
- LIMA: Less Is More for Alignment 引用了 Training language models to follow instructions with human feedback
- Fast Data Attribution for Text-to-Image Models 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- Training language models to follow instructions with human feedback 引用了 Language Models are Few-Shot Learners
- IF-Guide: Influence Function-Guided Detoxification of LLMs 引用了 Training language models to follow instructions with human feedback
引用它的顶会 Paper6
- Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data SchedulerZixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei 等NeurIPS 2025 · 16 次引用
- Better Training Data Attribution via Better Inverse Hessian-Vector ProductsAndrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae 等NeurIPS 2025 · 12 次引用
- IF-Guide: Influence Function-Guided Detoxification of LLMsZachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun HongNeurIPS 2025 · 9 次引用
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence EstimationDmytro Vitel, Anshuman ChhabraICLR 2026 · 8 次引用
- Fast Data Attribution for Text-to-Image ModelsSheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Richard Zhang 等NeurIPS 2025 · 7 次引用
它引用的 Paper18
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 24,707 次引用
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 次引用
- LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer 等NeurIPS 2023 · 1,486 次引用
相关 Paper
- Understanding Impact of Human Feedback via Influence FunctionsTaywon Min, Haeone Lee, Yongchan Kwon, Kimin LeeACL 2025 · 11 次引用
- RRInf: Efficient Influence Function Estimation via Ridge Regression for Large Language Models and Text-to-Image Diffusion ModelsZhuozhuo Tu, Cheng Chen, Yuxuan DuEMNLP 2025
- In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language ModelsAyrton San Joaquin, Bin Wang, Zhengyuan Liu, Philippe Muller 等EMNLP Findings 2024 · 3 次引用
- Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning ModelsAnshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra 等ICML 2025
- EULoInf: Efficient Hessian-Free Entropy Based Uncertainty-Aware Data Influence ApproximationRunxin Cai, Jingtan Wang, Bryan Kian Hsiang LowACL Findings 2026
