Lune

CVPR Findings2026

Background-Compensated Audio-Visual Semantic Modulation Framework for Audio-Visual Event Localization

Chao Sun, Junbo Zhang, Chuanbo Zhu, Mingjun Huang, Bo Du

Abstract

Audio-visual event localization (AVEL) aims to locate and classify video segments where visual and audio signals are semantically related. While audio-visual correlations are often modeled through interactions and fusion based on single-modal features, background information is typically suppressed as noise. However, audio-visual events (AVEs) are inherently tied to their surrounding context, which serves as a semantic bridge between visual and audio modalities, containing latent cross-modal semantic cues. To address this, we generate and refine frame-level AVE background captions using multi-modal large language models. We then propose an efficient Background-Compensated Audio-Visual Semantic Modulation (BC-AVSMod) framework for AVEL. It introduces a semantic alignment module based on vision-language pretraining models, marking the first use of background captions to enhance cross-modal interactions in AVEL. Specialized Adapters are also employed to achieve efficient semantic-aligned feature embedding. Furthermore, a semantic calibration and fusion module with cross-modal semantic modulation layers is proposed to capture audio-visual correlations. Extensive experiments on the AVE dataset demonstrate that BC-AVSMod achieves state-of-the-art performance under both fully and weakly supervised settings. Ablation studies further validate the effectiveness of our approach.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on19

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  3. ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations
  4. Attention Bottlenecks for Multimodal FusionArsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen et al.NeurIPS 2021 · 884 citations
  5. Dual Attention Matching for Audio-Visual Event LocalizationYu Wu, Linchao Zhu, Yan Yan, Yi YangICCV 2019 · 233 citations

Related papers

  1. Cross-modal Background Suppression for Audio-Visual Event LocalizationYan Xia, Zhou ZhaoCVPR 2022 · 62 citations
  2. Audio-Visual Semantic Graph Network for Audio-Visual Event LocalizationLiang Liu, Shuaiyong Li, Yongqiang ZhuCVPR 2025
  3. Learning Event-Specific Localization Preferences for Audio-Visual Event LocalizationShiping Ge, Zhiwei Jiang, Yafeng Yin, Cong Wang et al.ACM MM 2023 · 12 citations
  4. CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event LocalizationXiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao et al.ACM MM 2024 · 8 citations
  5. Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event CaptioningTanzila Rahman, Bicheng Xu, Leonid SigalICCV 2019 · 89 citations
Dusk over the sea between two cliffs drawn in fine vertical lines