Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization
Jiayu Xiong, Jing Wang, Qi Zhang, Wanlong Wang, Jun Xue
Abstract
Audio-visual deepfake localization demands interval-level outputs that serve as temporal evidence. Despite recent progress, symmetric fusion under single-sided or asynchronous forgeries propagates cross-modal noise, degrading high-precision localization. We present IaMSB, an inconsistency-aware multimodal Schrödinger Bridge (SB) that jointly estimates cross-modal consistency and performs interval-level localization. Unlike diffusion models, SB minimizes path-distribution discrepancy and yields consistency scores without explicit noise injection or denoising. With the Schrödinger Bridge (SB), IaMSB unifies consistency estimation, cross-modal information selection, and bridge-step scheduling in one framework. Specifically, a lightweight coarse bridge first proposes candidate intervals and estimates cross-modal consistency; these statistics select cross-modal witness signals and allocate bridge steps asymmetrically across modalities. A refinement bridge then performs step-tuned fusion and outputs refined, time-aligned intervals. IaMSB anticipates single-sided and asynchronous forgeries and, using bottlenecked cross-modal interaction with step allocation, suppresses noise transfer, avoids unnecessary iterations. Across benchmarks, IaMSB stabilizes strict-IoU boundary precision, raising AP@0.95 by , and yields improved high-precision localization, particularly for single-sided forgeries.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 0fe3467f-65ad-417c-be0a-47857f5d74abBuilds on20
- VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-TrainingZhan Tong, Yibing Song, Jue Wang, Limin WangNeurIPS 2022 · 2,336 citations
- Diffusion Schrödinger Bridge with Applications to Score-Based Generative ModelingValentin De Bortoli, James Thornton, Jeremy Heng, Arnaud DoucetNeurIPS 2021 · 811 citations
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster PredictionBowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman MohamedICLR 2022 · 460 citations
- Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and LocalizationKomal Chugh, Parul Gupta, Abhinav Dhall, Ramanathan SubramanianACM MM 2020 · 217 citations
- MAViL: Masked Audio-Video LearnersPo-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali et al.NeurIPS 2023 · 95 citations
Related papers
- Structural–Semantic Perception for Diffusion-Guided Temporal Forgery LocalizationLigong Cao, Yeting Guo, Haoang ChiCVPR 2026
- Intra-Modal and Cross-Modal Synchronization for Audio-Visual Deepfake Detection and Temporal LocalizationAshutosh Anshul, Shreyas Gopal, Deepu Rajan, Eng Siong ChngICCV 2025 · 10 citations
- A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery LocalizationWenbo Xu, Junyan Wu, Wei Lu, Xiangyang Luo et al.ACM MM 2025 · 2 citations
- Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media VerificationZizhao Chen, Ping Wei, Ziyang Ren, Huan Li et al.CVPR 2026
- Frequency-Dependent Scheduled Schrödinger Bridge for Underwater Acoustic Signal DenoisingPengsen Zhu, Lina Gao, Yulong Huang, Lifeng Liu et al.AAAI 2026
