Explainable AI for Issue Classification: A Multi-class Study with LIME and SHAP

Issue classification is a fundamental task in software development, enabling teams to manage issue reports. Automatic issue classification can help developers classify issue reports. However, developers should understand why each issue report is classified in such a way. A prior study has shown that explainable AI (XAI) can explain how an issue report is classified as a bug or a non-bug. However, the binary setting limits applicability to real-world issue tracking systems, where multiple categories coexist. In this paper, we replicate and extend the prior study by conducting a multi-class issue classification experiment using three categories: Bug, Enhancement, and Question. We use a fine-tuned, seBERT-based classifier and apply two widely used XAI models, LIME and SHAP, to generate explanations for issue classification. We then analyze the results of applying LIME and SHAP to multi-class issue classification, both qualitatively and quantitatively.