Lune

EMNLP Findings2025

SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models

Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee, Radu Marinescu, Katsiaryna Mirylenka, Michael R. Glass, Xiao Shou

Abstract

When does a large language model (LLM) know what it does not know? Uncertainty quantification (UQ) provides measures of uncertainty, such as an estimate of the confidence in an LLM's generated output, and is therefore increasingly recognized as a crucial component of trusted AI systems. Black-box UQ methods do not require access to internal model information from the generating LLM and therefore have numerous real-world advantages, such as robustness to system changes, adaptability to choice of LLM, reduced costs, and computational tractability. In this paper, we investigate the effectiveness of UQ techniques that are primarily but not necessarily entirely blackbox, where the consistency between a generated output and other sampled generations is used as a proxy for confidence in its correctness. We propose a high-level non-verbalized similarity-based aggregation framework that subsumes a broad swath of UQ approaches suitable for complex generative tasks, as well as introduce specific novel techniques from the framework that train confidence estimation models using small training sets. Through an empirical study with datasets spanning the diverse tasks of question answering, summarization, and text-to-SQL, we demonstrate that our proposed similarity-based methods can yield better calibrated confidences than baselines.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on19

  1. Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister et al.NeurIPS 2023 · 1,549 citations
  2. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionMohammadreza Pourreza, Davood RafieiNeurIPS 2023 · 909 citations
  3. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li et al.ICLR 2024 · 867 citations
  4. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  5. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language ModelsPotsawee Manakul, Adian Liusie, Mark J. F. GalesEMNLP 2023 · 331 citations

Related papers

  1. Confidence Estimation for Text-to-SQL in Large Language ModelsSepideh Entezari Maleki, Mohammadreza Pourreza, Davood RafieiAAAI 2026 · 1 citation
  2. CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMsRoman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii et al.NeurIPS 2025 · 33 citations
  3. Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty QuantificationKimia Hamidieh, Veronika Thost, Walter Gerych, Mikhail Yurochkin et al.ICLR 2026 · 12 citations
  4. ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage GuaranteesZhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang et al.EMNLP Findings 2024 · 16 citations
  5. Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam SearchEkaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin et al.ICLR 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines