Learning Deep Attribution Priors Based On Prior Knowledge

Feature attribution methods, which explain an individual prediction made by a model as a sum of attributions for each input feature, are an essential tool for understanding the behavior of complex deep learning models. However, ensuring that models produce meaningful explanations, rather than ones that rely on noise, is not straightforward. Exacerbating this problem is the fact that attribution methods do not provide insight as to why features are assigned their attribution values, leading to explanations that are difficult to interpret. In real-world problems we often have sets of additional information for each feature that are predictive of that feature's importance to the task at hand. Here, we propose the deep attribution prior (DAPr) framework to exploit such information to overcome the limitations of attribution methods. Our framework jointly learns a relationship between prior information and feature importance, as well as biases models to have explanations that rely on features predicted to be important. We find that our framework both results in networks that generalize better to out of sample data and admits new methods for interpreting model behavior. Recent advances in machine learning have come in the form of complex models that humans struggle to interpret. In response to the black-box nature of these models, a variety of recent work has focused on model interpretability [11] . One particular line of work that has gained much attention is that of feature attribution methods [22, 37, 28, 3, 33] . Given a model and a specific prediction made by that model, these methods assign a numeric value to each input feature, indicating how important that feature was for the given prediction (Figure 1a ). A variety of such methods have been proposed, and previous work has focused on how such methods can be used to gain insight into model behavior in applications where model trust is critical [43, 31] Given a set of attributions, a natural question is why a feature was assigned a specific attribution value. In some settings it is easy for a human to evaluate the sensibility of attributions; for example, in image classification problems we can overlay attribution values on the original image. However, in many other domains we do not have the ability to assess the validity of attribution values so 34th Conference on Neural Information Processing Systems (NeurIPS 2020),