ACL Findings2022
Calibration of Machine Reading Systems at Scale
Shehzaad Dhuliawala, Leonard Adolphs, Rajarshi Das, Mrinmaya Sachan
Abstract
In typical machine learning systems, an estimate of the probability of the prediction is used to assess the system's confidence in the prediction. This confidence measure is usually uncalibrated; i.e. the system's confidence in the prediction does not match the true probability of the predicted output. In this paper, we present an investigation into calibrating open setting machine reading systems such as open-domain question answering and claim verification systems. We show that calibrating such complex systems which contain a discrete retrieval and deep reading components is challenging and current calibration techniques fail to scale to these settings. We propose simple extensions to existing calibration approaches that allow us to adapt these callibrators to these settings. Our experimental results reveal that the joint callibration of the retriever and the reader outperforms the reader calibrator by a significant margin. We also show that the callibrator can be useful for selective prediction, e.g., when question answering systems are posed with unanswerable or out-ofthe-training distribution questions.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering cites Dense Passage Retrieval for Open-Domain Question Answering
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering cites Distilling Knowledge from Reader to Retriever for Question Answering
- Distilling Knowledge from Reader to Retriever for Question Answering cites Dense Passage Retrieval for Open-Domain Question Answering
- A Diachronic Perspective on User Trust in AI under Uncertainty cites Selective Question Answering under Domain Shift
Cited by top-tier papers3
- The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language ModelsAviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan et al.EMNLP 2023 · 11 citations
- A Diachronic Perspective on User Trust in AI under UncertaintyShehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, Mrinmaya SachanEMNLP 2023 · 7 citations
- Bridging Personalization and Control in Scientific Personalized SearchSheshera Mysore, Garima Dhanania, Kishor Patil, Surya Kallumadi et al.SIGIR 2025 · 1 citation
Builds on7
- Distilling Knowledge from Reader to Retriever for Question AnsweringGautier Izacard, Edouard GraveICLR 2021 · 317 citations
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question AnsweringDevendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer et al.NeurIPS 2021 · 197 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- Selective Question Answering under Domain ShiftAmita Kamath, Robin Jia, Percy LiangACL 2020 · 121 citations
- Calibrating Structured Output Predictors for Natural Language ProcessingAbhyuday Jagannatha, Hong YuACL 2020 · 22 citations
Related papers
- Knowing More About Questions Can Help: Improving Calibration in Question AnsweringShujian Zhang, Chengyue Gong, Eunsol ChoiACL Findings 2021 · 25 citations
- Re-Examining Calibration: The Case of Question AnsweringChenglei Si, Chen Zhao, Sewon Min, Jordan L. Boyd-GraberEMNLP Findings 2022 · 10 citations
- When to Read Documents or QA History: On Unified and Selective Open-domain QAKyungjae Lee, Sang-eun Han, Seung-won Hwang, Moontae LeeACL Findings 2023 · 4 citations
- End-to-End Training of Neural Retrievers for Open-Domain Question AnsweringDevendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant et al.ACL 2021
- Phrase Retrieval for Open Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive LearningSoyeong Jeong, Jinheon Baek, Sung Ju Hwang, Jong ParkACL Findings 2023 · 6 citations
