CREPE: Open-Domain Question Answering with False Presuppositions

When asking about unfamiliar topics, information seeking users often pose questions with false presuppositions. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct. This is in large part due to difficulty in retrieving relevant evidence passages from a large text corpus. CREPE provides a benchmark to study question answering in the wild, and our analyses provide avenues for future work in better modeling and further studying the task. 1 Question: If there's an equal and opposite reaction for everything, how does any action happen? Isn't it balanced out by the opposite reaction? False presupposition: The equal and opposite reaction applies to the same object. Correction: Based on Newton's Law of Motion, the equal and opposite reaction applies to the other object. Only forces that are applied to the same object would be cancelled out. Newton's laws of motion From Wikipedia, the free encyclopedia Inputs given to the human raters Question: Why do prosecuters/courts seek/sentence prison time greater than the expected lifespan of the offender (i.e. 150 years in prison)? Why not simply sentence those criminals to 'life' in prison instead? Comment: Sentencing options are written into state laws. Life in prison is different in state laws than 150 years. Some of it comes into play with the "cruel and unusual punishment" clause in the Constitution too. Life in prison may not be "cruel and unusual" for a murder sentence, but it might be for, say, child sex trafficking. But if you trafficked 10 kids and the sentence is 15 years for each one, you get an effective life sentence that will also stand up, Constitutionally, against a "cruel and unusual punishment" defense. Outputs human raters rate Reference Presupposition: It does not make sense to sentence a person to 150 years in prison if they can't live that long anyways, prosecutors should use the life in prison sentence instead. Correction: The defendant can argue the life in prison sentence as cruel and unusual, so the actual year sentence is better to give than the alternative. GOLD-COMMENT track, Dedicated Presupposition: Penalties should be able to be sentenced to life in prison. Correction: Life in prison is different in state laws than 150 years in prison. GOLD-COMMENT track, Unified Presupposition: If a criminal is sentenced to life in prison, they should be sentenced to life in prison. Correction: It is not the case that if a criminal is sentenced to life in prison, they should be sentenced to life in prison. Main, Dedicated Presupposition: Penalties should be able to be imposed on criminals for life. Correction: The longer the sentence, the more likely the prosecution will seek to sentence the offender to life in prison. Main, Unified Presupposition: Prosecutor's should seek prison time greater than the expected lifespan of the offender. Correction: It is not the case that prosecutor's should seek prison time greater than the expected lifespan of the offender. Table 13 : An example of the input and the output human raters are given for the human evaluation of the writing subtask. Note that human raters are not given which output is a reference or from which system. Inputs given to the human raters Question: Why did scientists in the 1970s think that there was going to be a new ice age soon? Comment: They didn't. Between 1965 and 1979, there was 7 papers talking about global cooling (not ice age and not necessarily soon). During the same period there was 44 papers about global warming. The media just liked the sensationalism, so there was some news article and a front page on the Times Magazine. They started with a minority of scientist talking about global cooling in a time period when there was still a lot of unknown in climate science and changed that to Scientific consensus that an Ice Age is coming soon.