Second-Order Uncertainty Quantification: A Distance-Based Approach
Yusuf Sale, Viktor Bengs, Michele Caprio, Eyke Hüllermeier
Abstract
In the past couple of years, various approaches to representing and quantifying different types of predictive uncertainty in machine learning, notably in the setting of classification, have been proposed on the basis of second-order probability distributions, i.e., predictions in the form of distributions on probability distributions. A completely conclusive solution has not yet been found, however, as shown by recent criticisms of commonly used uncertainty measures associated with second-order distributions, identifying undesirable theoretical properties of these measures. In light of these criticisms, we propose a set of formal criteria that meaningful uncertainty measures for predictive uncertainty based on second-order distributions should obey. Moreover, we provide a general framework for developing uncertainty measures to account for these criteria, and offer an instantiation based on the Wasserstein distance, for which we prove that all criteria are satisfied.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Integral Imprecise Probability Metrics cites Credal Learning Theory
- Probabilistic Self-supervised Representation Learning via Scoring Rules Minimization cites On Second-Order Scoring Rules for Epistemic Uncertainty Quantification
- The Unreasonable Effectiveness of Deep Evidential Regression cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
- On Second-Order Scoring Rules for Epistemic Uncertainty Quantification cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
- On Second-Order Scoring Rules for Epistemic Uncertainty Quantification cites The Unreasonable Effectiveness of Deep Evidential Regression
- On Second-Order Scoring Rules for Epistemic Uncertainty Quantification cites Deep Evidential Regression
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites The Unreasonable Effectiveness of Deep Evidential Regression
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites On Second-Order Scoring Rules for Epistemic Uncertainty Quantification
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Deep Evidential Regression
- Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
- Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs cites On Second-Order Scoring Rules for Epistemic Uncertainty Quantification
Cited by top-tier papers10
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?Mira Jürgens, Nis Meinert, Viktor Bengs, Eyke Hüllermeier et al.ICML 2024 · 35 citations
- Credal Learning TheoryMichele Caprio, Maryam Sultana, Eleni Elia, Fabio CuzzolinNeurIPS 2024 · 34 citations
- Experts Don't Cheat: Learning What You Don't Know By Predicting PairsDaniel D. Johnson, Daniel Tarlow, David Duvenaud, Chris J. MaddisonICML 2024 · 18 citations
- Integral Imprecise Probability MetricsSiu Lun Chau, Michele Caprio, Krikamol MuandetNeurIPS 2025 · 15 citations
- Probabilistic Self-supervised Representation Learning via Scoring Rules MinimizationAmirhossein Vahidi, Simon Schoßer, Lisa Wimmer, Yawei Li et al.ICLR 2024 · 6 citations
Builds on5
- Deep Evidential RegressionAlexander Amini, Wilko Schwarting, Ava Soleimany, Daniela RusNeurIPS 2020 · 777 citations
- Pitfalls of Epistemic Uncertainty Quantification through Loss MinimisationViktor Bengs, Eyke Hüllermeier, Willem WaegemanNeurIPS 2022 · 78 citations
- On Uncertainty, Tempering, and Data Augmentation in Bayesian ClassificationSanyam Kapoor, Wesley J. Maddox, Pavel Izmailov, Andrew Gordon WilsonNeurIPS 2022 · 64 citations
- The Unreasonable Effectiveness of Deep Evidential RegressionNis Meinert, Jakob Gawlikowski, Alexander LavinAAAI 2023 · 58 citations
- On Second-Order Scoring Rules for Epistemic Uncertainty QuantificationViktor Bengs, Eyke Hüllermeier, Willem WaegemanICML 2023 · 37 citations
Related papers
- Uncertainty Quantification for Machine Learning: One Size Does Not Fit AllPaul Hofman, Yusuf Sale, Eyke HüllermeierAAAI 2026 · 2 citations
- Rethinking Aleatoric and Epistemic UncertaintyFreddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark van der Wilk et al.ICML 2025
- Nonparametric Uncertainty Quantification for Single Deterministic Neural NetworkNikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov et al.NeurIPS 2022 · 52 citations
- Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised LearningTakuo MatsubaraNeurIPS 2024 · 7 citations
- Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss PredictionsJake Snell, Thomas P. Zollo, Zhun Deng, Toniann Pitassi et al.ICLR 2023 · 1 citation
