Theoretical and Practical Perspectives on what Influence Functions Do

Influence functions (IF) have been seen as a technique for explaining model predictions through the lens of the training data. Their utility is assumed to be in identifying training examples "responsible" for a prediction so that, for example, correcting a prediction is possible by intervening on those examples (removing or editing them) and retraining the model. However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-andretrain effect poorly. In order to understand the mismatch between the theoretical promise and the practical results, we analyse five assumptions made by IF methods which are problematic for modern-scale deep neural networks and which concern convexity, numeric stability, training trajectory and parameter divergence. This allows us to clarify what can be expected theoretically from IF. We show that while most assumptions can be addressed successfully, the parameter divergence poses a clear limitation on the predictive power of IF: influence fades over training time even with deterministic training. We illustrate this theoretical result with BERT and ResNet models. Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples. Recently, in their extensive experiments, [BPF21] and [KS21] could not find empirical support for the claim that IF approximate the Leave-Some-Out Retraining (LSOR) effect on the loss in deep neural networks. In particular, they show that the correlation between the ranking of training examples produced by LSOR and the IF-based ranking is low and considerably affected by choice of hyperparameters. How can this discrepancy be explained? And, given that the theoretical justification for IF lacks empirical support, does it mean that IF should be abandoned as an explainability and debugging tool altogether? In this work we clarify what question Influence Functions (IF) actually answer. We first identify assumptions which are either implicit or not investigated in prior work: these concern convexity, Preprint. Under review.