Testing Your Question Answering Software via Asking Recursively

CHAPTER 23 Question Answering The quest for knowledge is deeply human, and so it is not surprising that practically as soon as there were computers we were asking them questions. By the early 1960s, systems used the two major paradigms of question answering-informationretrieval-based and knowledge-based-to answer questions about baseball statistics or scientific facts. Even imaginary computers got into the act. Deep Thought, the computer that Douglas Adams invented in The Hitchhiker's Guide to the Galaxy, managed to answer "the Ultimate Question Of Life, The Universe, and Everything". 1 In 2011, IBM's Watson question-answering system won the TV game-show Jeopardy!, surpassing humans at answering questions like: WILLIAM WILKINSON'S "AN ACCOUNT OF THE PRINCIPALITIES OF WALLACHIA AND MOLDOVIA" INSPIRED THIS AUTHOR'S MOST FAMOUS NOVEL 2 Question answering systems are designed to fill human information needs that might arise in situations like talking to a virtual assistant, interacting with a search engine, or querying a database. Most question answering systems focus on a particular subset of these information needs: factoid questions, questions that can be answered with simple facts expressed in short texts, like the following: (23.1) Where is the Louvre Museum located? (23.2) What is the average age of the onset of autism? In this chapter we describe the two major paradigms for factoid question answering. Information-retrieval (IR) based QA, sometimes called open domain QA, relies on the vast amount of text on the web or in collections of scientific papers like PubMed. Given a user question, information retrieval is used to find relevant passages. Then neural reading comprehension algorithms read these retrieved passages and draw an answer directly from spans of text. In the second paradigm, knowledge-based question answering, a system instead builds a semantic representation of the query, such as mapping What states border Texas? to the logical representation: λ x.state(x) ∧ borders(x,texas), or When was Ada Lovelace born? to the gapped relation: birth-year (Ada Lovelace, ?x). These meaning representations are then used to query databases of facts. We'll also briefly discuss two other QA paradigms. We'll see how to query a language model directly to answer a question, relying on the fact that huge pretrained language models have already encoded a lot of factoids. And we'll sketch classic pre-neural hybrid question-answering algorithms that combine information from IRbased and knowledge-based sources. We'll explore the possibilities and limitations of all these approaches, along the way also introducing two technologies that are key for question answering but also 1 The answer was 42, but unfortunately the details of the question were never revealed. 2 The answer, of course, is 'Who is Bram Stoker', and the novel was Dracula.