Succinct Interaction-Aware Explanations

Shapley values (Shap) are a popular approach to explaining decisions of black-box models by revealing the importance of individual features. Shap explanations are easy to interpret, but as they do not incorporate feature interactions, they are also incomplete and potentially misleading. Interaction-aware methods such as nShap report the additive importance of all subsets up to n features. These explanations are complete, but in practice excessively large and difficult to interpret. In this paper, we combine the best of both worlds. We partition the features into significantly interacting groups, and use these to compose a succinct, interpretable explanation. To determine which partitioning out of super-exponentially many explains a model best, we derive a criterion that weighs the complexity of an explanation against its representativeness for the model's behavior. To be able to find the best partitioning, we show how to prune sub-optimal solutions using a statistical test. This not only improves runtime but also helps to avoid explaining spurious interactions. Experiments show that iShap represents underlying modeling more accurately than Shap and nShap, and a user study suggests that iShap is perceived as more interpretable and trustworthy.