EMNLP Findings2023
QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering
Haochen Shi, Weiqi Wang, Tianqing Fang, Baixuan Xu, Wenxuan Ding, Xin Liu, Yangqiu Song
Abstract
Zero-shot commonsense Question-Answering (QA) requires models to reason about general situations beyond specific benchmarks.Stateof-the-art approaches fine-tune language models on QA pairs constructed from Common-Sense Knowledge Bases (CSKBs) to equip the models with more commonsense knowledge in a QA context.However, current QA synthesis protocols may introduce noise from the CSKBs and generate ungrammatical questions and false negative options, which impede the model's ability to generalize.To address these issues, we propose QADYNAM-ICS, a training dynamics-driven framework for QA diagnostics and refinement.Our approach analyzes the training dynamics of each QA pair at both the question level and option level, discarding machine-detectable artifacts by removing uninformative QA pairs and mislabeled or false-negative options.Extensive experiments demonstrate the effectiveness of our approach, which outperforms all baselines while using only 33% of the synthetic data, even including LLMs such as ChatGPT.Moreover, expert evaluations confirm that our framework significantly improves the quality of QA synthesis.Our code and model checkpoints are available at https://github.com/HKUST- KnowComp/QaDynamics. Question Candidates Confidence MISLABELED OPTIONFlynn is cleaning out Flynn's garage.As a result, Flynn felt A: confused.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? cites Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning cites PIQA: Reasoning about Physical Commonsense in Natural Language
- CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning cites Language Models are Few-Shot Learners
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers2
- CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense ReasoningWeiqi Wang, Tianqing Fang, Chunyang Li, Haochen Shi et al.ACL 2024 · 10 citations
- Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?Ning Bian, Xianpei Han, Hongyu Lin, Yaojie Lu et al.ACL 2024 · 1 citation
Builds on24
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
Related papers
- CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question AnsweringWeiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu et al.EMNLP Findings 2023 · 19 citations
- ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question AnsweringFrancesco Molfese, Simone Conia, Riccardo Orlando, Roberto NavigliEMNLP 2024
- Zero-Shot Commonsense Question Answering with Cloze Translation and Consistency OptimizationZi-Yi Dou, Nanyun PengAAAI 2022 · 29 citations
- Leveraging QA Datasets to Improve Generative Data AugmentationDheeraj Mekala, Tu Vu, Timo Schick, Jingbo ShangEMNLP 2022 · 8 citations
- Generating then Refining for Reliable Knowledge Base Question AnsweringJianqi Gao, Hang Yu, Jian Cao, Ranran Bu et al.ACL 2026
