Towards Distribution-shift Robust Text Classification of Emotional Content

Supervised models based on Transformers have been shown to achieve impressive performances in many natural language processing tasks. However, besides requiring a large amount of costly manually annotated data, supervised models tend to adapt to the characteristics of the training dataset, which are usually created ad-hoc and whose data distribution often differs from the one in real applications, showing significant performance degradation in real-world scenarios. We perform an extensive assessment of the out-of-distribution performances of supervised models for classification in the emotion and hate-speech detection tasks and show that NLI-based zeroshot models often outperform them, making task-specific annotation useless when the characteristics of final-user data are not known in advance. To benefit from both supervised and zero-shot approaches, we propose to finetune an NLI-based model on the task-specific dataset. The resulting model often outperforms all available supervised models both in distribution and out of distribution, with only a few thousand training samples. Recent zero-shot models (Yin et al., 2019; Liu et al., 2021) have gained popularity thanks to their ability to reduce the dependency on task-specific annotated data by enabling models to predict previously unseen labels. For instance, models trained for Next Sentence Prediction (NLP) or Natural Language Inference (NLI) tasks can be applied to infer