Zero-Shot Emotion Recognition via Affective Structural Embedding

Image emotion recognition attracts much attention in recent years due to its wide applications. It aims to understand the emotional response of humans, where candidate emotion categories are generally defined by specific psychological theories. However, with the development of psychological theories, emotion categories become increasingly diverse, fine-grained, and difficult to collect samples. In this paper, we investigate zero-shot learning (ZSL) problem in the emotion recognition task, which aims to recognize the new unseen emotions. Specifically, we propose an affective structural embedding framework, utilizing mid-level semantic representation, i.e., adjective-noun pairs (ANP) features, to construct an intermediate embedding space. By doing this, the learned intermediate space can bridge the affective gap between low-level visual features and high-level semantics. In addition, we introduce an adversarial constraint to combine the visual and affective embeddings so as to retain the discriminative capacity of visual features and the affective structural information of semantic features during training process. Our method is evaluated on five widelyused affective datasets and the experimental results show that the proposed algorithm outperforms the state-of-theart approaches.