One Explanation is Not Enough: Structured Attention Graphs for Image Classification

Saliency maps are popular tools for explaining the decisions of convolutional neural networks (CNNs) for image classification. Typically, for each image of interest, a single saliency map is produced, which assigns weights to pixels based on their importance to the classification. We argue that a single saliency map provides an incomplete understanding since there are often many other maps that can explain a classification equally well. In this paper, we propose to utilize a beam search algorithm to systematically search for multiple explanations for each image. Results show that there are indeed multiple relatively localized explanations for many images. However, naively showing multiple explanations to users can be overwhelming and does not reveal their common and distinct structures. We introduce structured attention graphs (SAGs), which compactly represent sets of attention maps for an image by visualizing how different combinations of image regions impact the confidence of a classifier. An approach to computing a compact and representative SAG for visualization is proposed via diverse sampling. We conduct a user study comparing the use of SAGs to traditional saliency maps for answering comparative counterfactual questions about image classifications. Our results show that user accuracy is increased significantly when presented with SAGs compared to standard saliency map baselines.