EMNLP Findings2023
Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach
Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, Natthawut Kertkeidkachorn
Abstract
Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose a novel method for identifying shortcut reasoning. The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning. Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on6
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet ClassificationFrancesco Barbieri, José Camacho-Collados, Luis Espinosa Anke, Leonardo NevesEMNLP Findings 2020 · 720 citations
- Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
- Explaining Black Box Predictions and Unveiling Data Artifacts through Influence FunctionsXiaochuang Han, Byron C. Wallace, Yulia TsvetkovACL 2020 · 91 citations
- Beyond Accuracy: Behavioral Testing of NLP Models with CheckListMarco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer SinghACL 2020 · 51 citations
- Combining Feature and Instance Attribution to Detect ArtifactsPouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron C. WallaceACL Findings 2022 · 27 citations
Related papers
- Finding Dataset Shortcuts with Grammar InductionDan Friedman, Alexander Wettig, Danqi ChenEMNLP 2022 · 5 citations
- HuaSLIM: Human Attention Motivated Shortcut Learning Identification and Mitigation for Large Language modelsYuqi Ren, Deyi XiongACL Findings 2023 · 4 citations
- Improving the robustness of NLI models with minimax trainingMichalis Korakakis, Andreas VlachosACL 2023 · 4 citations
- Towards Faithful Explanations: Boosting Rationalization with Shortcuts DiscoveryLinan Yue, Qi Liu, Yichao Du, Li Wang et al.ICLR 2024 · 10 citations
- Think Faster Than Words: Efficient LLM Chain-of-Thought Reasoning via Dynamic Shortcut DecodingFan Liu, Yanhao Wang, Min Zhang, Zhikang Chen et al.ACL 2026
