Lune

ICLR2025Top-tier venue

Provable Uncertainty Decomposition via Higher-Order Calibration

Gustaf Ahdritz, Aravind Gollakota, Parikshit Gopalan, Charlotte Peale, Udi Wieder

Abstract

We give a principled method for decomposing the predictive uncertainty of a model into aleatoric and epistemic components with explicit semantics relating them to the real-world data distribution. While many works in the literature have proposed such decompositions, they lack the type of formal guarantees we provide. Our method is based on the new notion of higher-order calibration, which generalizes ordinary calibration to the setting of higher-order predictors that predict mixtures over label distributions at every point. We show how to measure as well as achieve higher-order calibration using access to kk-snapshots, namely examples where each point has kk independent conditional labels. Under higher-order calibration, the estimated aleatoric uncertainty at a point is guaranteed to match the real-world aleatoric uncertainty averaged over all points where the prediction is made. To our knowledge, this is the first formal guarantee of this type that places no assumptions whatsoever on the real-world data distribution. Importantly, higher-order calibration is also applicable to existing higher-order predictors such as Bayesian and ensemble models and provides a natural evaluation metric for such models. We demonstrate through experiments that our method produces meaningful uncertainty decompositions for image classification.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Selective Omniprediction and Fair AbstentionSílvia Casacuberta, Varun KanadeNeurIPS 2025 · 3 citations

Builds on9

  1. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance AwarenessJeremiah Z. Liu, Zi Lin, Shreyas Padhy, Dustin Tran et al.NeurIPS 2020 · 604 citations
  2. Uncertainty Estimation Using a Single Deep Deterministic Neural NetworkJoost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin GalICML 2020 · 529 citations
  3. Uncertainty Estimation in Autoregressive Structured PredictionAndrey Malinin, Mark J. F. GalesICLR 2021 · 439 citations
  4. Human Uncertainty Makes Classification More RobustJoshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, Olga RussakovskyICCV 2019 · 362 citations
  5. Pitfalls of Epistemic Uncertainty Quantification through Loss MinimisationViktor Bengs, Eyke Hüllermeier, Willem WaegemanNeurIPS 2022 · 78 citations

Related papers

  1. From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian EstimationNikita Kotelevskii, Vladimir Kondratyev, Martin Takác, Eric Moulines et al.ICLR 2025
  2. On Second-Order Scoring Rules for Epistemic Uncertainty QuantificationViktor Bengs, Eyke Hüllermeier, Willem WaegemanICML 2023 · 37 citations
  3. Experts Don't Cheat: Learning What You Don't Know By Predicting PairsDaniel D. Johnson, Daniel Tarlow, David Duvenaud, Chris J. MaddisonICML 2024 · 18 citations
  4. Nonparametric Uncertainty Quantification for Single Deterministic Neural NetworkNikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov et al.NeurIPS 2022 · 52 citations
  5. Uncertainty-Aware Deep Classifiers Using Generative ModelsMurat Sensoy, Lance M. Kaplan, Federico Cerutti, Maryam SalekiAAAI 2020 · 88 citations
Dusk over the sea between two cliffs drawn in fine vertical lines