Learning model uncertainty as variance-minimizing instance weights

Predictive uncertainty-a model's self-awareness regarding its accuracy on an inputis key for both building robust models via training interventions and for test-time applications such as selective classification. We propose a novel instance-conditional reweighting approach that captures predictive uncertainty using an auxiliary network, and unifies these train-and test-time applications. The auxiliary network is trained using a meta-objective in a bilevel optimization framework. A key contribution of our proposal is the meta-objective of minimizing dropout variance, an approximation of Bayesian predictive uncertainty, We show in controlled experiments that we effectively capture diverse specific notions of uncertainty through this meta-objective, while previous approaches only capture certain aspects. These results translate to significant gains in real-world settings-selective classification, label noise, domain adaptation, calibration-and across datasets-Imagenet, Cifar100, diabetic retinopathy, Camelyon, WILDs, Imagenet-C,-A,-R, Clothing1M, etc. For Diabetic Retinopathy, we see upto 3.4%/3.3% accuracy AUC gains over SOTA in selective classification. We also improve upon large-scale pretrained models such as PLEX (Tran et al., 2022).

Published as a conference paper at ICLR 2024

The question that motivates our work is: Given training and validation set, what is the best reweighing function of training instances that yields a good uncertainty measure at test time? Further, what robustness properties are achieved by the reweighted classifier?

We propose a novel instance dependent weight learning algorithm for learning predictive uncertainty -REVAR(Reweighting for Dropout Variance Reduction). We propose to learn an auxiliary uncertainty model p = g(x) (which we call U-SCORE) alongside training of the primary model y = f (x). U-SCORE unifies train and test-time applications of uncertainty. Our primary algorithmic insight is the use of a novel dropout based variance regularization term in the U-SCORE objective. Below we summarize our approach and key contributions.