FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

One type of question that is commonly found in day-to-day scenarios is "fan-out" questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over interdocument dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation. 1 1 https://fanoutqa.com https://github.com/zhudotexe/fanoutqa What is the total number of employees in the five largest banks in the world? How many employees does HDFC Bank have? JPMorgan Chase, Bank of America, ..., HDFC Bank