MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking

Fact-checking real-world claims often requires reviewing multiple multimodal documents to assess a claim's truthfulness, which is a highly laborious and time-consuming task. In this paper, we present a summarization model designed to generate claim-specific summaries useful for fact-checking from multimodal, multi-document datasets. The model takes inputs in the form of documents, images, and a claim, with the objective of assisting in fact-checking tasks. We introduce a dynamic perceiver-based model that can handle inputs from multiple modalities of arbitrary lengths. To train our model, we leverage a novel reinforcement learning-based entailment objective to generate summaries that provide evidence distinguishing between different truthfulness labels. To assess the efficacy of our approach, we conduct experiments on both an existing benchmark and a new dataset of multi-document claims that we contribute. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and demonstrates strong performance on our new Multi-News-Fact-Checking dataset. Claim: Adhering to COVID-19 precautions, such as mask-wearing and encouraging breastfeeding, can be crucial for safeguarding young children, as studies suggest protective antibodies in breast milk may offer potential protection against COVID-19, despite the rarity of severe infections in newborns. … . Existing evidence indicates that COVID-19 infections in newborns are uncommon, but they do happen. Transmission is primarily thought to occur when babies are exposed to family members or caregivers infected with COVID-19 themselves. New parents can take comfort in the fact that most babies with COVID-19 experience asymptomatic infections or mild disease. … … . Breast milk is encouraged whether through breastfeeding or pumping, and protective antibodies are in breastmilk. Studies are still underway to determine if the antibodies protect the baby from COVID-19. However, similar antibody levels from other vaccines, like whooping cough, administered during pregnancy have proven to be protective for babies. … Generated summary for fact checking: For young, unvaccinated children, adherence to proven COVID-19 precautions, including mask-wearing in public, is essential to minimize infection risks and potential complications. Encouraging breastfeeding, with its protective antibodies, may offer defense against COVID-19, akin to the proven efficacy of antibodies from other vaccines administered during pregnancy. …