Lune

ACL Findings2021

Parallel Attention Network with Sequence Matching for Video Grounding

Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

Abstract

Given a video, video grounding aims to retrieve a temporal moment that semantically corresponds to a language query. In this work, we propose a Parallel Attention Network with Sequence matching (SeqPAN) to address the challenges in this task: multi-modal representation learning, and target moment boundary prediction. We design a self-guided parallel attention module to effectively capture selfmodal contexts and cross-modal attentive information between video and text. Inspired by sequence labeling tasks in natural language processing, we split the ground truth moment into begin, inside, and end regions. We then propose a sequence matching strategy to guide start/end boundary predictions using region labels. Experimental results on three datasets show that SeqPAN is superior to state-of-theart methods. Furthermore, the effectiveness of the self-guided parallel attention module and the sequence matching module is verified. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Partial Annotation-based Video Moment Retrieval via Iterative Learning cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
  • Partial Annotation-based Video Moment Retrieval via Iterative Learning cites Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localization
  • Partial Annotation-based Video Moment Retrieval via Iterative Learning cites Span-based Localizing Network for Natural Language Video Localization
  • HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training cites VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  • HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training cites SlowFast Networks for Video Recognition
  • Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
  • Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites Span-based Localizing Network for Natural Language Video Localization
  • Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites SlowFast Networks for Video Recognition
  • Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
  • Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localization cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
  • Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localization cites Span-based Localizing Network for Natural Language Video Localization
  • MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
  • MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction cites Span-based Localizing Network for Natural Language Video Localization
  • SnAG: Scalable and Accurate Video Grounding cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
  • SnAG: Scalable and Accurate Video Grounding cites Span-based Localizing Network for Natural Language Video Localization
  • SnAG: Scalable and Accurate Video Grounding cites SlowFast Networks for Video Recognition
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment RetrievalZhihang Liu, Jun Li, Hongtao Xie, Pandeng Li et al.AAAI 2024 · 49 citations
  2. Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video LocalizationDaizong Liu, Wei HuACM MM 2022 · 33 citations
  3. Partial Annotation-based Video Moment Retrieval via Iterative LearningWei Ji, Renjie Liang, Lizi Liao, Hao Fei et al.ACM MM 2023 · 17 citations
  4. MS-DETR: Natural Language Video Localization with Sampling Moment-Moment InteractionJing Wang, Aixin Sun, Hao Zhang, Xiaoli LiACL 2023 · 13 citations
  5. SnAG: Scalable and Accurate Video GroundingFangzhou Mu, Sicheng Mo, Yin LiCVPR 2024 · 13 citations

Builds on15

  1. SlowFast Networks for Video RecognitionChristoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming HeICCV 2019 · 4,104 citations
  2. VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
  3. Learning 2D Temporal Adjacent Networks for Moment Localization with Natural LanguageSongyang Zhang, Houwen Peng, Jianlong Fu, Jiebo LuoAAAI 2020 · 579 citations
  4. HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingLinjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan et al.EMNLP 2020 · 387 citations
  5. Span-based Localizing Network for Natural Language Video LocalizationHao Zhang, Aixin Sun, Wei Jing, Joey Tianyi ZhouACL 2020 · 279 citations

Related papers

  1. Negative Sample Matters: A Renaissance of Metric Learning for Temporal GroundingZhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li et al.AAAI 2022 · 170 citations
  2. Relation-aware Video Reading Comprehension for Temporal Language GroundingJialin Gao, Xin Sun, Mengmeng Xu, Xi Zhou et al.EMNLP 2021 · 51 citations
  3. Temporally Grounding Language Queries in Videos by Contextual Boundary-Aware PredictionJingwen Wang, Lin Ma, Wenhao JiangAAAI 2020 · 206 citations
  4. Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text UnderstandingJongbhin Woo, Hyeonggon Ryu, Youngjoon Jang, Jae-Won Cho et al.ACM MM 2024 · 3 citations
  5. Cascaded Prediction Network via Segment Tree for Temporal Video GroundingYang Zhao, Zhou Zhao, Zhu Zhang, Zhijie LinCVPR 2021
Dusk over the sea between two cliffs drawn in fine vertical lines