Finding Dataset Shortcuts with Grammar Induction
摘要
Many NLP datasets have been found to contain shortcuts: simple decision rules that achieve surprisingly high accuracy. However, it is difficult to discover shortcuts automatically. Prior work on automatic shortcut detection has focused on enumerating features like unigrams or bigrams, which can find only low-level shortcuts, or relied on post-hoc model interpretability methods like saliency maps, which reveal qualitative patterns without a clear statistical interpretation. In this work, we propose to use probabilistic grammars to characterize and discover shortcuts in NLP datasets. Specifically, we use a contextfree grammar to model patterns in sentence classification datasets and use a synchronous context-free grammar to model datasets involving sentence pairs. The resulting grammars reveal interesting shortcut features in a number of datasets, including both simple and high-level features, and automatically identify groups of test examples on which conventional classifiers fail. Finally, we show that the features we discover can be used to generate diagnostic contrast examples and incorporated into standard robust optimization methods to improve worst-group accuracy. 1
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 3 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Evaluating Models' Local Decision Boundaries via Contrast Sets 引用了 Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
- Grammar Prompting for Domain-Specific Language Generation with Large Language Models 引用了 Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammars
- Explore Spurious Correlations at the Concept Level in Language Models for Text Classification 引用了 Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
- Explore Spurious Correlations at the Concept Level in Language Models for Text Classification 引用了 Just Train Twice: Improving Group Robustness without Training Group Information
引用它的顶会 Paper3
- Grammar Prompting for Domain-Specific Language Generation with Large Language ModelsBailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao 等NeurIPS 2023 · 138 次引用
- Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive GrammarsSonglin Yang, Roger Levy, Yoon KimACL 2023 · 1 次引用
- Explore Spurious Correlations at the Concept Level in Language Models for Text ClassificationYuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An 等ACL 2024
它引用的 Paper10
- Just Train Twice: Improving Group Robustness without Training Group InformationEvan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan 等ICML 2021 · 683 次引用
- Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 次引用
- Understanding Global Feature Contributions With Additive Importance MeasuresIan Covert, Scott M. Lundberg, Su-In LeeNeurIPS 2020 · 476 次引用
- Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant 等EMNLP Findings 2020 · 260 次引用
- Explaining Black Box Predictions and Unveiling Data Artifacts through Influence FunctionsXiaochuang Han, Byron C. Wallace, Yulia TsvetkovACL 2020 · 91 次引用
相关 Paper
- Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free ApproachDaichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, Natthawut KertkeidkachornEMNLP Findings 2023
- Improving the robustness of NLI models with minimax trainingMichalis Korakakis, Andreas VlachosACL 2023 · 4 次引用
- Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context LearningRuixiang Tang, Dehan Kong, Longtao Huang, Hui XueACL Findings 2023 · 31 次引用
- Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language ModelsYuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei ZhuEMNLP Findings 2024 · 3 次引用
- Causally motivated multi-shortcut identification and removalJiayun Zheng, Maggie MakarNeurIPS 2022 · 27 次引用
