Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention

Multimodal summarization requires models to jointly understand textual and visual inputs to generate concise, semantically coherent summaries. Existing methods often inject shallow visual features into deep language models, leading to representational mismatches and weak cross-modal grounding. We propose a unified framework that jointly performs text summarization and representative image selection. Our system, SPeCTrA-Sum (Sampler Perceiver with Cross-modal Transformer and gated Attention for Summarization), introduces two key innovations. First, a Deep Visual Processor (DVP) aligns the visual encoder with the language model at corresponding depths, enabling hierarchical, layer-wise fusion that preserves semantic consistency. Second, a lightweight Visual Relevance Predictor (VRP) selects salient and diverse images by distilling soft labels from a Determinantal Point Process (DPP) teacher. SPeCTrA-Sum is trained using a multi-objective loss that combines autoregressive summarization, cross-modal alignment, and DPP-based distillation. Experiments show that our system produces more accurate, visually grounded summaries and selects more representative images, demonstrating the benefits of depth-aware fusion and principled image selection for multimodal summarization. * Corresponding Author A record spring heat wave hit Sydney in early October, painting the harbour skyline in hazy orange hues. Emergency services warned that tinder-dry bushland across New South Wales was primed for bushfires, and small blazes quickly flared along the coast. Scientists noted that these extremes are consistent with ongoing climate change and argued that a rapid shift to renewable energy, such as wind farms, could help curb future risks. Meanwhile, many residents followed the unfolding situation through digital news apps, sharing updates with neighbours and community groups. Sydney faces an intense spring heat wave and bushfires, spurring calls for renewable energy. Residents rely on digital news apps to stay informed.