Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model

The current paradigm of training deep neural networks for classification tasks includes minimizing the empirical risk, pushing the training loss value towards zero even after the training classification error has vanished. In this terminal phase of training, it has been observed that the last-layer features collapse to their class-means and these class-means converge to the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is termed as Neural Collapse (N C). However, this characterization only holds in classbalanced datasets where every class has the same number of training samples. When the training dataset is class-imbalanced, some N C properties will no longer hold true, for example, the geometry of class-means will skew away from the simplex ETF. In this paper, we generalize N C to imbalanced regime for cross-entropy loss under the unconstrained ReLU features model. We demonstrate that while the within-class features collapse property still holds in this setting, the class-means will converge to a structure consisting of orthogonal vectors with lengths dependent on the number of training samples. Furthermore, we find that the classifier weights (i.e., the last-layer linear classifier) are aligned to the scaled and centered class-means, with scaling factors dependent on the number of training samples of each class. This generalizes N C in the class-balanced setting. We empirically validate our results through experiments on practical architectures and dataset.