Answering Open-Domain Questions of Varying Reasoning Steps from Text

We develop a unified system to answer directly from text open-domain questions that may require a varying number of retrieval steps. We employ a single multi-task transformer model to perform all the necessary subtasks-retrieving supporting facts, reranking them, and predicting the answer from all retrieved documents-in an iterative fashion. We avoid crucial assumptions of previous work that do not transfer well to real-world settings, including exploiting knowledge of the fixed number of retrieval steps required to answer each question or using structured metadata like knowledge bases or web links that have limited availability. Instead, we design a system that can answer open-domain questions on any text collection without prior knowledge of reasoning complexity. To emulate this setting, we construct a new benchmark, called B QA, by combining existing one-and twostep datasets with a new collection of 530 questions that require three Wikipedia pages to answer, unifying Wikipedia corpora versions in the process. We show that our model demonstrates competitive performance on both existing benchmarks and this new benchmark. We make the new benchmark available at https: //beerqa.github.io/. The Lord of the Rings à "150 million copies" Q. The Ingerophrynus Gollum is named after a character in a book that sold how many copies? Retriever Reader A. 150 million copies Answer exists in one of the reasoning paths No answer exist Repeat N times until the answer found is confident enough Reranker Query Generator Expand reasoning path with top-ranked paragraph … WIKIPEDIA search ① Q à "Ingerophrynus Gollum" ④ Q + Ingerophrynus Gollum à "Lord of the Rings" ② Q + retrieved paras à NOANSWER ⑤ Q + Ingerophrynus Gollum + The Lord of the Rings à "150 million copies"