EMNLP Findings2022
Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes
Abstract
Multi-hop Question Answering (QA) is a challenging task since it requires an accurate aggregation of information from multiple context paragraphs and a thorough understanding of the underlying reasoning chains.Recent work in multi-hop QA has shown that performance can be boosted by first decomposing the questions into simpler, single-hop questions.In this paper, we explore one additional utility of the multi-hop decomposition from the perspective of explainable NLP: to create explanation by probing a neural QA model with them.We hypothesize that in doing so, users will be better able to predict when the underlying QA system will give the correct answer.Through human participant studies, we verify that exposing the decomposition probes and answers to the probes to users can increase their ability to predict system performance on a question instance basis.We show that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation.In-depth analyses show the need for improvements in decomposition systems. 1 How do I know it has the right answer?What year did the band that sang "With or Without You" form?1976 U2 Which band sang "With or Without You"?What year did U2 form?1976 I think the agent knows what it is talking about
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA SystemsNavita Goyal, Eleftheria Briakou, Amanda Liu, Connor Baumler et al.EMNLP 2023 · 4 citations
- Measuring User's Mental Models of Speech Translation in Human-AI CollaborationHyojung Han, Nishant Balepur, Jordan Lee Boyd-Graber, Marine CarpuatACL 2026
Builds on4
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Peter Hase, Mohit BansalACL 2020 · 216 citations
- Select, Answer and Explain: Interpretable Multi-Hop Reading Comprehension over Multiple DocumentsMing Tu, Kevin Huang, Guangtao Wang, Jing Huang et al.AAAI 2020 · 155 citations
- Unsupervised Question Decomposition for Question AnsweringEthan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho et al.EMNLP 2020 · 6 citations
Related papers
- Latent Reasoning for Low-Resource Question GenerationXinting Huang, Jianzhong Qi, Yu Sun, Rui ZhangACL Findings 2021 · 5 citations
- Answering Questions by Meta-Reasoning over Multiple Chains of ThoughtOri Yoran, Tomer Wolfson, Ben Bogin, Uri Katz et al.EMNLP 2023 · 30 citations
- Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-AnsweringHarsh Jhamtani, Peter ClarkEMNLP 2020 · 2 citations
- Chain-of-Question: A Progressive Question Decomposition Approach for Complex Knowledge Base Question AnsweringYixing Peng, Quan Wang, Licheng Zhang, Yi Liu et al.ACL Findings 2024 · 4 citations
- Multi-hop Question Generation with Graph Convolutional NetworkDan Su, Yan Xu, Wenliang Dai, Ziwei Ji et al.EMNLP Findings 2020 · 39 citations
