EMNLP Findings2022
Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering
Xuan Luo, Chuang Fan, Yice Zhang, Wanguo Jiang, Bing Qin, Ruifeng Xu
Abstract
We propose a simple ranking strategy to solve a generative commonsense question answering (QA) problem. Compared with multiple-choice QA, it is challenging because the answers to a question are not unique and they are supposed to be popular and diverse. Our strategy exploits the dataset itself and negative samples that we collect from WordNet to train a ranker that picks out the most popular answers for commonsense questions. The effectiveness of our strategy is verified on different pre-trained masked language models (MLMs) in a pipeline framework, where an MLM reranks the generated answers. Further, we explore an endto-end framework where MLMs are utilized to guide the generation of generative language models (GLMs). Taking advantage of reinforcement learning, we apply policy gradient to train a GLM with the rewards fed back by an MLM. Empirical results on ProtoQA dataset demonstrate that MLMs can acquire the ability to distinguish the popular answers and improve the typical answer generation of GLMs as well.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on9
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPengcheng He, Jianfeng Gao, Weizhu ChenICLR 2023 · 394 citations
- Evaluating Commonsense in Pre-Trained Language ModelsXuhui Zhou, Yue Zhang, Leyang Cui, Dandan HuangAAAI 2020 · 198 citations
- Pre-training Is (Almost) All You Need: An Application to Commonsense ReasoningAlexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot et al.ACL 2020 · 30 citations
Related papers
- Zero-Shot Commonsense Question Answering with Cloze Translation and Consistency OptimizationZi-Yi Dou, Nanyun PengAAAI 2022 · 29 citations
- ExpertPLM: Pre-training Expert Representation for Expert FindingQiyao Peng, Hongtao LiuEMNLP Findings 2022 · 8 citations
- CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language ModelsChenhao Wang, Jiachun Li, Yubo Chen, Kang Liu et al.EMNLP 2022 · 2 citations
- A Semantic-based Method for Unsupervised Commonsense Question AnsweringYilin Niu, Fei Huang, Jiaming Liang, Wenkai Chen et al.ACL 2021
- Generated Knowledge Prompting for Commonsense ReasoningJiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck et al.ACL 2022
