A Proposal-Based Approach for Activity Image-to-Video Retrieval
Ruicong Xu, Li Niu, Jianfu Zhang, Liqing Zhang
Abstract
Activity image-to-video retrieval task aims to retrieve videos containing the similar activity as the query image, which is a challenging task because videos generally have many background segments irrelevant to the activity. In this paper, we utilize R-C3D model to represent a video by a bag of activity proposals, which can filter out background segments to some extent. However, there are still noisy proposals in each bag. Thus, we propose an Activity Proposal-based Image-to-Video Retrieval (APIVR) approach, which incorporates multi-instance learning into cross-modal retrieval framework to address the proposal noise issue. Specifically, we propose a Graph Multi-Instance Learning (GMIL) module with graph convolutional layer, and integrate this module with classification loss, adversarial loss, and triplet loss in our cross-modal retrieval framework. Moreover, we propose geometry-aware triplet loss based on point-to-subspace distance to preserve the structural information of activity proposals. Extensive experiments on three widely-used datasets verify the effectiveness of our approach.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers5
- HANet: Hierarchical Alignment Networks for Video-Text RetrievalPeng Wu, Xiangteng He, Mingqian Tang, Yiliang Lv et al.ACM MM 2021 · 62 citations
- Activity Image-to-Video Retrieval by Disentangling Appearance and MotionLiu Liu, Jiangtong Li, Li Niu, Ruicong Xu et al.AAAI 2021 · 29 citations
- Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and TransformerMinyi Zhao, Jinpeng Wang, Dongliang Liao, Yiru Wang et al.SIGIR 2023 · 3 citations
- Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product RetrievalXiaowan Hu, Yiyi Chen, Yan Li, Minquan Wang et al.ACM MM 2024 · 1 citation
- Multimodal Fusion via Self-Consistent Task-Gradient FieldsJiayu Xiong, Jing Wang, Jun Xue, Wanlong Wang et al.ICML 2026
Related papers
- C3CMR: Cross-Modality Cross-Instance Contrastive Learning for Cross-Media RetrievalJunsheng Wang, Tiantian Gong, Zhixiong Zeng, Changchang Sun et al.ACM MM 2022 · 12 citations
- Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video RetrievalCheol-Ho Cho, WonJun Moon, Woojin Jun, Minseok Jung et al.AAAI 2025 · 11 citations
- Multi-Modal Relational Graph for Cross-Modal Video Moment RetrievalYawen Zeng, Da Cao, Xiaochi Wei, Meng Liu et al.CVPR 2021
- Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed VideosJunwen Chen, Wentao Bao, Yu KongACM MM 2020 · 19 citations
- Visual Co-Occurrence Alignment Learning for Weakly-Supervised Video Moment RetrievalZheng Wang, Jingjing Chen, Yu-Gang JiangACM MM 2021 · 74 citations
