Lune

AAAI2020Top-tier venue

A Proposal-Based Approach for Activity Image-to-Video Retrieval

Ruicong Xu, Li Niu, Jianfu Zhang, Liqing Zhang

Abstract

Activity image-to-video retrieval task aims to retrieve videos containing the similar activity as the query image, which is a challenging task because videos generally have many background segments irrelevant to the activity. In this paper, we utilize R-C3D model to represent a video by a bag of activity proposals, which can filter out background segments to some extent. However, there are still noisy proposals in each bag. Thus, we propose an Activity Proposal-based Image-to-Video Retrieval (APIVR) approach, which incorporates multi-instance learning into cross-modal retrieval framework to address the proposal noise issue. Specifically, we propose a Graph Multi-Instance Learning (GMIL) module with graph convolutional layer, and integrate this module with classification loss, adversarial loss, and triplet loss in our cross-modal retrieval framework. Moreover, we propose geometry-aware triplet loss based on point-to-subspace distance to preserve the structural information of activity proposals. Extensive experiments on three widely-used datasets verify the effectiveness of our approach.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multimodal Fusion via Self-Consistent Task-Gradient Fields cites Activity Image-to-Video Retrieval by Disentangling Appearance and Motion
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. HANet: Hierarchical Alignment Networks for Video-Text RetrievalPeng Wu, Xiangteng He, Mingqian Tang, Yiliang Lv et al.ACM MM 2021 · 62 citations
  2. Activity Image-to-Video Retrieval by Disentangling Appearance and MotionLiu Liu, Jiangtong Li, Li Niu, Ruicong Xu et al.AAAI 2021 · 29 citations
  3. Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and TransformerMinyi Zhao, Jinpeng Wang, Dongliang Liao, Yiru Wang et al.SIGIR 2023 · 3 citations
  4. Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product RetrievalXiaowan Hu, Yiyi Chen, Yan Li, Minquan Wang et al.ACM MM 2024 · 1 citation
  5. Multimodal Fusion via Self-Consistent Task-Gradient FieldsJiayu Xiong, Jing Wang, Jun Xue, Wanlong Wang et al.ICML 2026

Related papers

  1. C3CMR: Cross-Modality Cross-Instance Contrastive Learning for Cross-Media RetrievalJunsheng Wang, Tiantian Gong, Zhixiong Zeng, Changchang Sun et al.ACM MM 2022 · 12 citations
  2. Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video RetrievalCheol-Ho Cho, WonJun Moon, Woojin Jun, Minseok Jung et al.AAAI 2025 · 11 citations
  3. Multi-Modal Relational Graph for Cross-Modal Video Moment RetrievalYawen Zeng, Da Cao, Xiaochi Wei, Meng Liu et al.CVPR 2021
  4. Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed VideosJunwen Chen, Wentao Bao, Yu KongACM MM 2020 · 19 citations
  5. Visual Co-Occurrence Alignment Learning for Weakly-Supervised Video Moment RetrievalZheng Wang, Jingjing Chen, Yu-Gang JiangACM MM 2021 · 74 citations
Dusk over the sea between two cliffs drawn in fine vertical lines