Can NLI Models Verify QA Systems' Predictions?

To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pretrained models and recent prior datasets to construct powerful question conversion and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to evaluate QA systems' proposed answers. We show that our approach improves the confidence estimation of a QA model across different domains. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, i.e., when the answer sentence does not address all aspects of the question. Context: The first season of the fantasy comedy television series The Good Place, created by Michael Schur, aired … The series focuses on Eleanor Shellstrop (Kristen Bell) , a woman who wakes up in the afterlife and is introduced by Michael (Ted Danson) to a Heaven-like utopia he designed … Question: Who plays the bad guy in the Good Place? Answer: Ted Danson Premise: The series The Good Place focuses on Eleanor Shellstrop (Kristen Bell) , a woman who wakes up in the afterlife and is introduced by Michael (Ted Danson) to a Heaven-like utopia he designed. Hypothesis: Ted Danson plays the bad guy in The Good Place.

Decontextualization of the answer sentence Question conversion to a declarative statement

Answer is correct, but information about Michael being the bad guy is missing in the premise

Not entailed, answer rejected