Cross-modal Ambiguity Learning for Multimodal Fake News Detection

Cross-modal learning is essential to enable accurate fake news detection due to the fast-growing multimodal contents in online social communities. A fundamental challenge of multimodal fake news detection lies in the inherent ambiguity across different content modalities, i.e., decisions made from unimodalities may disagree with each other, which may lead to inferior multimodal fake news detection. To address this issue, we formulate the cross-modal ambiguity learning problem from an information-theoretic perspective and propose CAFE -an ambiguity-aware multimodal fake news detection method. CAFE mainly consists of 1) a cross-modal alignment module to transform the heterogeneous unimodality features into a shared semantic space, 2) a cross-modal ambiguity learning module to estimate the ambiguity between different modalities and 3) a cross-modal fusion module to capture the cross-modal correlations. Based on such design, CAFE can judiciously and adaptively aggregate unimodal features and cross-modal correlations, i.e., rely on unimodal features when cross-modal ambiguity is weak and refer to cross-modal correlations when cross-modal ambiguity is strong, to achieve more accurate fake news detection. Experimental studies on two widely used datasets (Twitter and Weibo) demonstrate that CAFE can outperform state-of-the-art fake news detection methods by 2.2-18.9% and 1.7-11.4% in terms of accuracy, respectively.