Lune

ICML2023Top-tier venue

On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

Viktor Bengs, Eyke Hüllermeier, Willem Waegeman

Abstract

It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

15 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
  • Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? cites Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?
  • Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? cites Deep Evidential Regression
  • Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? cites Ensemble Distribution Distillation
  • Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? cites Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
  • Evidential Deep Learning for Open Set Action Recognition cites Deep Evidential Regression
  • Evidential Deep Learning for Open Set Action Recognition cites Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
  • Second-Order Uncertainty Quantification: A Distance-Based Approach cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
  • Second-Order Uncertainty Quantification: A Distance-Based Approach cites Deep Evidential Regression
  • Conformalized Credal Set Predictors cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
  • Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation cites Ensemble Distribution Distillation
  • Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation cites Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
  • Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Evidential Deep Learning for Open Set Action Recognition
  • Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Second-Order Uncertainty Quantification: A Distance-Based Approach
  • Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
  • Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Deep Evidential Regression
  • Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? cites Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
  • Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs cites Second-Order Uncertainty Quantification: A Distance-Based Approach
  • Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs cites Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
  • Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts cites Ensemble Distribution Distillation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers15

  1. Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?Mira Jürgens, Nis Meinert, Viktor Bengs, Eyke Hüllermeier et al.ICML 2024 · 35 citations
  2. Second-Order Uncertainty Quantification: A Distance-Based ApproachYusuf Sale, Viktor Bengs, Michele Caprio, Eyke HüllermeierICML 2024 · 34 citations
  3. Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?Maohao Shen, Jongha Jon Ryu, Soumya Ghosh, Yuheng Bu et al.NeurIPS 2024 · 29 citations
  4. Conformalized Credal Set PredictorsAlireza Javanmardi, David Stutz, Eyke HüllermeierNeurIPS 2024 · 28 citations
  5. Experts Don't Cheat: Learning What You Don't Know By Predicting PairsDaniel D. Johnson, Daniel Tarlow, David Duvenaud, Chris J. MaddisonICML 2024 · 18 citations

Builds on11

  1. Deep Evidential RegressionAlexander Amini, Wilko Schwarting, Ava Soleimany, Daniela RusNeurIPS 2020 · 777 citations
  2. Ensemble Distribution DistillationAndrey Malinin, Bruno Mlodozeniec, Mark J. F. GalesICLR 2020 · 273 citations
  3. Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-CountsBertrand Charpentier, Daniel Zügner, Stephan GünnemannNeurIPS 2020 · 263 citations
  4. Evidential Deep Learning for Open Set Action RecognitionWentao Bao, Qi Yu, Yu KongICCV 2021 · 204 citations
  5. Pitfalls of Epistemic Uncertainty Quantification through Loss MinimisationViktor Bengs, Eyke Hüllermeier, Willem WaegemanNeurIPS 2022 · 78 citations

Related papers

  1. Uncertainty Quantification for Machine Learning: One Size Does Not Fit AllPaul Hofman, Yusuf Sale, Eyke HüllermeierAAAI 2026 · 2 citations
  2. Provable Uncertainty Decomposition via Higher-Order CalibrationGustaf Ahdritz, Aravind Gollakota, Parikshit Gopalan, Charlotte Peale et al.ICLR 2025
  3. Rethinking Aleatoric and Epistemic UncertaintyFreddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark van der Wilk et al.ICML 2025
  4. Possibilistic Predictive Uncertainty for Deep LearningYao Ni, Jeremie Houssineau, Yew Soon ONG, Piotr KoniuszICML 2026 · 2 citations
  5. Epistemic Uncertainty Quantification To Improve Decisions From Black-Box ModelsSébastien Melo, Gaël Varoquaux, Marine Le MorvanICLR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines