Lune

ICLR2021Top-tier venue

Predicting Classification Accuracy When Adding New Unobserved Classes

Yuli Slavutsky, Yuval Benjamini

Abstract

Multiclass classifiers are often designed and evaluated only on a sample from the classes on which they will eventually be applied. Hence, their final accuracy remains unknown. In this work we study how a classifier's performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes. For this, we define a measure of separation between correct and incorrect classes that is independent of the number of classes: the reversed ROC (rROC), which is obtained by replacing the roles of classes and data-points in the common ROC. We show that the classification accuracy is a function of the rROC in multiclass classifiers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added. Using these results we formulate a robust neural-network-based algorithm, CleaneX, which learns to estimate the accuracy of such classifiers on arbitrarily large sets of classes. Unlike previous methods, our method uses both the observed accuracies of the classifier and densities of classification scores, and therefore achieves remarkably better predictions than current state-of-the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Class Distribution Shifts in Zero-Shot Learning: Learning Robust RepresentationsYuli Slavutsky, Yuval BenjaminiNeurIPS 2024 · 5 citations

Related papers

  1. Confidence Calibration of Classifiers with Many ClassesAdrien Le-Coz, Stéphane Herbin, Faouzi AdjedNeurIPS 2024 · 21 citations
  2. What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?Weijian Deng, Stephen Gould, Liang ZhengICML 2021 · 72 citations
  3. Are Labels Always Necessary for Classifier Accuracy Evaluation?Weijian Deng, Liang ZhengCVPR 2021
  4. Prospective Representation Learning for Non-Exemplar Class-Incremental LearningWuxuan Shi, Mang YeNeurIPS 2024 · 9 citations
  5. Random-Set Neural NetworksShireen Kudukkil Manchingal, Muhammad Mubashar, Kaizheng Wang, Keivan Shariatmadar et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines