SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research

Despite its relevance, the maturity of NLP for social media pales in comparison with generalpurpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SUPERTWEETE-VAL, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SU-PERTWEETEVAL and our results suggest that, despite the recent advances in language modelling, social media remains challenging.