Lune

EMNLP Findings2022

Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering

Xuan Luo, Chuang Fan, Yice Zhang, Wanguo Jiang, Bing Qin, Ruifeng Xu

Abstract

We propose a simple ranking strategy to solve a generative commonsense question answering (QA) problem. Compared with multiple-choice QA, it is challenging because the answers to a question are not unique and they are supposed to be popular and diverse. Our strategy exploits the dataset itself and negative samples that we collect from WordNet to train a ranker that picks out the most popular answers for commonsense questions. The effectiveness of our strategy is verified on different pre-trained masked language models (MLMs) in a pipeline framework, where an MLM reranks the generated answers. Further, we explore an endto-end framework where MLMs are utilized to guide the generation of generative language models (GLMs). Taking advantage of reinforcement learning, we apply policy gradient to train a GLM with the rewards fed back by an MLM. Empirical results on ProtoQA dataset demonstrate that MLMs can acquire the ability to distinguish the popular answers and improve the typical answer generation of GLMs as well.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  3. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPengcheng He, Jianfeng Gao, Weizhu ChenICLR 2023 · 394 citations
  4. Evaluating Commonsense in Pre-Trained Language ModelsXuhui Zhou, Yue Zhang, Leyang Cui, Dandan HuangAAAI 2020 · 198 citations
  5. Pre-training Is (Almost) All You Need: An Application to Commonsense ReasoningAlexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot et al.ACL 2020 · 30 citations

Related papers

  1. Zero-Shot Commonsense Question Answering with Cloze Translation and Consistency OptimizationZi-Yi Dou, Nanyun PengAAAI 2022 · 29 citations
  2. ExpertPLM: Pre-training Expert Representation for Expert FindingQiyao Peng, Hongtao LiuEMNLP Findings 2022 · 8 citations
  3. CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language ModelsChenhao Wang, Jiachun Li, Yubo Chen, Kang Liu et al.EMNLP 2022 · 2 citations
  4. A Semantic-based Method for Unsupervised Commonsense Question AnsweringYilin Niu, Fei Huang, Jiaming Liang, Wenkai Chen et al.ACL 2021
  5. Generated Knowledge Prompting for Commonsense ReasoningJiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck et al.ACL 2022
Dusk over the sea between two cliffs drawn in fine vertical lines