Lune

EMNLP Findings2025

SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models

Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee, Radu Marinescu, Katsiaryna Mirylenka, Michael R. Glass, Xiao Shou

摘要

When does a large language model (LLM) know what it does not know? Uncertainty quantification (UQ) provides measures of uncertainty, such as an estimate of the confidence in an LLM's generated output, and is therefore increasingly recognized as a crucial component of trusted AI systems. Black-box UQ methods do not require access to internal model information from the generating LLM and therefore have numerous real-world advantages, such as robustness to system changes, adaptability to choice of LLM, reduced costs, and computational tractability. In this paper, we investigate the effectiveness of UQ techniques that are primarily but not necessarily entirely blackbox, where the consistency between a generated output and other sampled generations is used as a proxy for confidence in its correctness. We propose a high-level non-verbalized similarity-based aggregation framework that subsumes a broad swath of UQ approaches suitable for complex generative tasks, as well as introduce specific novel techniques from the framework that train confidence estimation models using small training sets. Through an empirical study with datasets spanning the diverse tasks of question answering, summarization, and text-to-SQL, we demonstrate that our proposed similarity-based methods can yield better calibrated confidences than baselines.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models 引用了 Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs 引用了 Self-Consistency Improves Chain of Thought Reasoning in Language Models
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper19

  1. Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister 等NeurIPS 2023 · 1,549 次引用
  2. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 次引用
  3. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li 等ICLR 2024 · 867 次引用
  4. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le 等ICLR 2023 · 681 次引用
  5. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language ModelsPotsawee Manakul, Adian Liusie, Mark J. F. GalesEMNLP 2023 · 331 次引用

相关 Paper

  1. Confidence Estimation for Text-to-SQL in Large Language ModelsSepideh Entezari Maleki, Mohammadreza Pourreza, Davood RafieiAAAI 2026 · 1 次引用
  2. CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMsRoman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii 等NeurIPS 2025 · 33 次引用
  3. Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty QuantificationKimia Hamidieh, Veronika Thost, Walter Gerych, Mikhail Yurochkin 等ICLR 2026 · 12 次引用
  4. ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage GuaranteesZhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang 等EMNLP Findings 2024 · 16 次引用
  5. Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam SearchEkaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin 等ICLR 2026 · 4 次引用
黄昏的海面,两侧是细线勾勒的悬崖