Action Dubber: Timing Audible Actions via Inflectional Flow
Wenlong Wan, Weiying Zheng, Tianyi Xiang, Guiqing Li, Shengfeng He
Abstract
We introduce the task of Audible Action Temporal Localization, which aims to identify the spatiotemporal coordinates of audible movements. Unlike conventional tasks such as action recognition and temporal action localization, which broadly analyze video content, our task focuses on the distinct kinematic dynamics of audible actions. It is based on the premise that key actions are driven by inflectional movements; for example, collisions that produce sound often involve abrupt changes in motion. To capture this, we propose T A 2 N et, a novel architecture that estimates inflectional flow using the second derivative of motion to determine collision timings without relying on audio input. T A 2 N et also integrates a self-supervised spatial localization strategy during training, combining contrastive learning with spatial analysis. This dual design improves temporal localization accuracy and simultaneously identifies sound sources within video frames. To support this task, we introduce a new benchmark dataset, Audible623, derived from Kinetics and UCF101 by removing non-essential vocalization subsets. Extensive experiments confirm the effectiveness of our approach on Audible623 and show strong generalizability to other domains, such as repetitive counting and sound source localization. Code and dataset are available at https:// github.com/WenlongWan/Audible623.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles cites Is Space-Time Attention All You Need for Video Understanding?
- Omnivore: A Single Model for Many Visual Modalities cites Is Space-Time Attention All You Need for Video Understanding?
Builds on19
- Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius, Heng Wang, Lorenzo TorresaniICML 2021 · 2,927 citations
- BMN: Boundary-Matching Network for Temporal Action Proposal GenerationTianwei Lin, Xiao Liu, Xin Li, Errui Ding et al.ICCV 2019 · 709 citations
- Self-Supervised Learning by Cross-Modal Audio-Video ClusteringHumam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani et al.NeurIPS 2020 · 483 citations
- Hiera: A Hierarchical Vision Transformer without the Bells-and-WhistlesChaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei et al.ICML 2023 · 388 citations
- Omnivore: A Single Model for Many Visual ModalitiesRohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten et al.CVPR 2022 · 185 citations
Related papers
- Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal SynchronicityPritam Sarkar, Ali EtemadAAAI 2023 · 16 citations
- Unsupervised Pre-training for Temporal Action Localization TasksCan Zhang, Tianyu Yang, Junwu Weng, Meng Cao et al.CVPR 2022 · 56 citations
- Weakly Supervised Temporal Action Localization Through Contrast Based Evaluation NetworksZiyi Liu, Le Wang, Qilin Zhang, Zhanning Gao et al.ICCV 2019 · 122 citations
- Learning Audio-Visual Source Localization via False Negative Aware Contrastive LearningWeixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu et al.CVPR 2023
- Telling Left From Right: Learning Spatial Correspondence of Sight and SoundKarren Yang, Bryan C. Russell, Justin SalamonCVPR 2020
