Adaspeaker: Learning Discriminative Speaker Representations with Gradient-Aware Adaptive Scaling

Learning discriminative representations of different speakers is a key challenge in open-set speaker recognition. To mitigate the mismatch between closed-set training and open-set testing, margin-based losses have been widely adopted to directly optimize the cosine similarity between speaker representations and proxy class vectors. While recent studies have shown that enhancing the margin for hard samples can improve representation learning, we observe three key limitations: (1) the measurement of sample hardness fails to fully capture differences in speaker representations, (2) margin-based emphasis does not significantly increase the gradient magnitude, and (3) the potential performance degradation caused by emphasizing hard samples are rarely considered. To address these issues, we propose Adaspeaker, a novel loss framework that combines an Intra-Inter sample hardness coefficient (Int2H) with a gradient-aware adaptive scaling strategy. Specifically, Int2H jointly models inter-class and intra-class hardness to estimate sample importance, which is subsequently used to adaptively scale cosine similarities for enhancing the gradient contribution of important samples. Experiments conducted on five evaluation settings show that Adaspeaker outperforms existing loss functions. Moreover, Adaspeaker can be seamlessly integrated into margin-based losses, yielding an average performance improvement of 12.6%. Code is available at https://github.com/LiuJinghan2001/Adaspeaker.