Identification of Multimodal Stance Towards Frames of Communication

Frames of communication are often evoked in multimedia documents. When an author decides to add an image to a text, one or both of the modalities may evoke a communication frame. Moreover, when evoking the frame, the author also conveys her/his stance towards the frame. Until now, determining if the author is in favor of, against or has no stance towards the frame was performed automatically only when processing texts. This is due to the absence of stance annotations on multimedia documents. In this paper we introduce MMVAX-STANCE, a dataset of 11,300 multimedia documents retrieved from social media, which have stance annotations towards 113 different frames of communication. This dataset allowed us to experiment with several models of multimedia stance detection, which revealed important interactions between texts and images in the inference of stance towards communication frames. When inferring the text/image relations, a set of 46,606 synthetic examples of multimodal documents with known stance was generated. This greatly impacted the quality of identifying multimedia stance, yielding an improvement of 20% in F1-score. Component Definition Examples of Frames of Communication Confidence Trust in the security and effectiveness of 2 Pfizer COVID-19 vaccine may cause anaphylaxis vaccinations, the health authorities, and in people with polyethylene glycol (PEG) allergy. the health officials who recommend 2 The Government has provided plenty of safety and develop vaccines. information about the COVID-19 vaccines. Complacency Complacency and laziness to get vaccinated 2 Preference for getting COVID-19 and fighting due to low perceived risk of infections. it off than vaccinating. Constraints Structural or psychological hurdles that 2 It takes courage both to vaccinate against make vaccination difficult or costly. COVID-19 and to refuse the vaccine. Calculation Degree to which personal costs and benefits 2 COVID-19 vaccines protect against the emerging of vaccination are weighted. variants. Collective Willingness to protect others and to 2 Vaccination is key in protecting yourself and others Responsibility eliminate infectious diseases. against COVID-19. Compliance Support for societal monitoring and sanctioning 2 People choosing not to get the COVID-19 vaccine of people who are not vaccinated. should not lose venue access/travel to some countries. Conspiracy Conspiracy thinking and belief in 2 COVID-19 vaccines make you 5G compatible. fake news related to vaccination. 2 The COVID vaccine renders pregnancies risky.