Lune

EMNLP Findings2025

Do Influence Functions Work on Large Language Models?

Zhe Li, Wei Zhao, Yige Li, Jun Sun

摘要

Influence functions are important for quantifying the impact of individual training data points on a model's predictions.Although extensive research has been conducted on influence functions in traditional machine learning models, their application to large language models (LLMs) has been limited.In this work, we conduct a systematic study to address a key question: do influence functions work on LLMs?Specifically, we evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings.Our further investigation reveals that their poor performance can be attributed to: (1) inevitable approximation errors when estimating the iHVP component due to the scale of LLMs, (2) uncertain convergence during fine-tuning, and, more fundamentally, (3) the definition itself, as changes in model parameters do not necessarily correlate with changes in LLM behavior.Thus, our study suggests the need for alternative approaches for identifying influential samples. The code is available at https://github.com/plumprc/ Failures-of-Influence-Functions-in-LLMs

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 6 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • LoRA: Low-Rank Adaptation of Large Language Models 引用了 Language Models are Few-Shot Learners
  • QLoRA: Efficient Finetuning of Quantized LLMs 引用了 LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs 引用了 Training language models to follow instructions with human feedback
  • Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler 引用了 LoRA: Low-Rank Adaptation of Large Language Models
  • First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation 引用了 LoRA: Low-Rank Adaptation of Large Language Models
  • First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation 引用了 QLoRA: Efficient Finetuning of Quantized LLMs
  • LIMA: Less Is More for Alignment 引用了 Training language models to follow instructions with human feedback
  • Fast Data Attribution for Text-to-Image Models 引用了 LoRA: Low-Rank Adaptation of Large Language Models
  • Training language models to follow instructions with human feedback 引用了 Language Models are Few-Shot Learners
  • IF-Guide: Influence Function-Guided Detoxification of LLMs 引用了 Training language models to follow instructions with human feedback
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper6

  1. Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data SchedulerZixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei 等NeurIPS 2025 · 16 次引用
  2. Better Training Data Attribution via Better Inverse Hessian-Vector ProductsAndrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae 等NeurIPS 2025 · 12 次引用
  3. IF-Guide: Influence Function-Guided Detoxification of LLMsZachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun HongNeurIPS 2025 · 9 次引用
  4. First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence EstimationDmytro Vitel, Anshuman ChhabraICLR 2026 · 8 次引用
  5. Fast Data Attribution for Text-to-Image ModelsSheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Richard Zhang 等NeurIPS 2025 · 7 次引用

它引用的 Paper18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 24,707 次引用
  3. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
  4. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 次引用
  5. LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer 等NeurIPS 2023 · 1,486 次引用

相关 Paper

  1. Understanding Impact of Human Feedback via Influence FunctionsTaywon Min, Haeone Lee, Yongchan Kwon, Kimin LeeACL 2025 · 11 次引用
  2. RRInf: Efficient Influence Function Estimation via Ridge Regression for Large Language Models and Text-to-Image Diffusion ModelsZhuozhuo Tu, Cheng Chen, Yuxuan DuEMNLP 2025
  3. In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language ModelsAyrton San Joaquin, Bin Wang, Zhengyuan Liu, Philippe Muller 等EMNLP Findings 2024 · 3 次引用
  4. Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning ModelsAnshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra 等ICML 2025
  5. EULoInf: Efficient Hessian-Free Entropy Based Uncertainty-Aware Data Influence ApproximationRunxin Cai, Jingtan Wang, Bryan Kian Hsiang LowACL Findings 2026
黄昏的海面,两侧是细线勾勒的悬崖