FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes

Real-world politically-opinionated memes often rely on figurative language to cloak propaganda and radical ideas to help them spread. It is not only a scientific challenge to develop machine learning models to recognize them in memes, but also sociologically beneficial to understand hidden meanings at scale and raise awareness. These memes are fast-evolving (in both topics and visuals) and it remains unclear whether current multimodal machine learning models are robust to such distribution shifts. To enable future research into this area, we first present FigMemes, a dataset for figurative language classification in politically-opinionated memes. 1 We evaluate the performance of state-of-the-art unimodal and multimodal models and provide comprehensive benchmark results. The key contributions of this proposed dataset include annotations of six commonly used types of figurative language in politicallyopinionated memes, and a wide range of topics and visual styles. We also provide analyses on the ability of multimodal models to generalize across distribution shifts in memes. Our dataset poses unique machine learning challenges and our results show that current models have significant room for improvement in both performance and robustness to distribution shifts. The code and dataset (including splits we used for analyses) are available at: https://github. com/UKPLab/emnlp2022-figmemes . * Equal Contributions. † Gregor is now affiliated with WüNLP & Computer Vision Lab, CAIDAS, University of Würzburg. 1 Disclaimer: The dataset contains racial slurs and other language/images that may be offensive to the readers. This dataset should only be used for academic research or noncommercial purposes.