Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples

Among existing uncertainty estimation approaches, Dirichlet Prior Network (DPN) distinctly models different predictive uncertainty types. However, for in-domain examples with high data uncertainties among multiple classes, even a DPN model often produces indistinguishable representations from the out-of-distribution (OOD) examples, compromising their OOD detection performance. We address this shortcoming by proposing a novel loss function for DPN to maximize the representation gap between in-domain and OOD examples. Experimental results demonstrate that our proposed approach consistently improves OOD detection performance. However, the true posterior of p(θ|D in ) is intractable. Hence, we need approximation such as Monte-Carlo dropout (MCDP) [11] , Langevin Dynamics [17], explicit ensembling [13]: p(ωc|x * , Din) ≈ where, θ (m) ∼ q(θ) is sampled from an explicit or implicit variational approximation, q(θ) of the true posterior p(θ|D in ). over class labels, given x * . Hence, the ensemble can be visualized as a collection of points on the probability simplex. For a confident prediction, it should be appeared sharply in one corner of the simplex. For an OOD example, it should be spread uniformly. We can determine the source of uncertainty in terms of the model uncertainty by measuring their spread. However, producing an ensemble distribution is computationally expensive. Further, it is difficult to control the desired behavior in practice [16] . Furthermore, for standard DNN models, with millions of parameters, it is even harder to find an appropriate prior distribution and the inference scheme to estimate the posterior distribution of the model. Few recent works, such as Dirichlet prior network (DPN) [5, 16], evidential deep learning (EDL) [18] etc, attempt to emulate this behavior by placing a Dirichlet distribution as a prior, over the predictive categorical distribution. In particular, DPN framework [5, 16] significantly improves the OOD detection performance by explicitly incorporating OOD training examples, D out , as we elaborate in Section 3. Non-Bayesian frameworks derive their measure of uncertainties using their predictive posteriors obtained from DNNs. Several works demonstrate that tweaking the input images using adversarial perturbations [19] can enhance the performance of a DNN for OOD detection [20, 21] . However, these approaches are sensitive to the tuning of parameters for each OOD distribution and difficult to apply for real-world applications. DeVries & Taylor (2018) [22] propose an auxiliary confidence estimation branch to derive OOD scores. Shalev et al. (2018) [23] use multiple semantic dense representations as the target label to train the OOD detection network. The works in [14, 15] also introduce multi-task loss, incorporating OOD data for training. Hein et al. (2019) [24] show that ReLU-networks lead to over-confident predictions even for samples that are far away from the in-domain distributions and propose methods to mitigate this problem [24] [25] [26] . While these models can identify the total predictive uncertainties, they cannot robustly determine whether the source of uncertainty is due to an in-domain data misclassification or an OOD example.