Pareto Deep Long-Tailed Recognition: A Conflict-Averse Solution

Deep long-tailed recognition (DLTR) has attracted much attention due to its close touch with realistic scenarios. Recent advances have focused on re-balancing across various aspects, e.g., sampling strategy, loss re-weighting, logit adjustment, and input/parameter perturbation, etc. However, few studies have considered dynamic re-balancing to address intrinsic optimization conflicts, which are identified as prevalent and critical issues in this study. In this paper, we empirically establish the severity of the optimization conflict issue in the DLTR scenario, which leads to a degradation of representation learning. This observation serves as the motivation for pursuing Pareto optimal solutions. Unfortunately, a straightforward integration of multi-objective optimization (MOO) with DLTR methods is infeasible due to the disparity between multi-task learning (MTL) and DLTR. Therefore, we propose effective alternatives by decoupling MOO-based MTL from a temporal perspective rather than a structural one. Furthermore, we enhance the integration of MOO and DLTR by investigating the generalization and convergence problems. Specifically, we propose optimizing the variability collapse loss, guided by the derived MOObased DLTR generalization bound, to improve generalization. Additionally, we anticipate worst-case optimization to ensure convergence. Building upon the proposed MOO framework, we introduce a novel method called Pareto deep LOng-Tailed recognition (PLOT). Extensive evaluations demonstrate that our method not only generally improves mainstream pipelines, but also achieves an augmented version to realize state-of-the-art performance across multiple benchmarks. Code is available at https://github.com/zzpustc/PLOT .