TextGenSHAP: Scalable Post-Hoc Explanations in Text Generation with Long Documents

Large language models (LLMs) have attracted great interest in many real-world applications; however, their "black-box" nature necessitates scalable and faithful explanations. Shapley values have matured as an explainability method for deep learning, but extending them to LLMs is difficult due to long input contexts and autoregressive output generation. We introduce TextGenSHAP, an efficient posthoc explanation method incorporating LLMspecific techniques, which leads to significant runtime improvements: token-level explanations in minutes not hours, and document-level explanations within seconds. We demonstrate how such explanations can improve end-toend performance of retrieval augmented generation by localizing important words within long documents and reranking passages collected by retrieval systems. On various opendomain question answering benchmarks, we show TextGenSHAP improves the retrieval recall and prediction accuracy significantly.