ViPLO: Vision Transformer Based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection
Jeeseung Park, Jin-Woo Park, Jong-Seok Lee
Abstract
Human-Object Interaction (HOI) detection, which localizes and infers relationships between human and objects, plays an important role in scene understanding. Although two-stage HOI detectors have advantages of high efficiency in training and inference, they suffer from lower performance than one-stage methods due to the old backbone networks and the lack of considerations for the HOI perception process of humans in the interaction classifiers. In this paper, we propose Vision Transformer based Pose-Conditioned Self-Loop Graph (ViPLO) to resolve these problems. First, we propose a novel feature extraction method suitable for the Vision Transformer backbone, called masking with overlapped area (MOA) module. The MOA module utilizes the overlapped area between each patch and the given region in the attention function, which addresses the quantization problem when using the Vision Transformer backbone. In addition, we design a graph with a pose-conditioned self-loop structure, which updates the human node encoding with local features of human joints. This allows the classifier to focus on specific human joints to effectively identify the type of interaction, which is motivated by the human perception process for HOI. As a result, ViPLO achieves the state-of-the-art results on two public benchmarks, especially obtaining a +2.07 mAP performance gain on the HICO-DET dataset. The source codes are available at https://github.com/Jeeseung- Park/ViPLO.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
29 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models cites Learning Transferable Visual Models From Natural Language Supervision
- Open-World Human-Object Interaction Detection via Multi-Modal Prompts cites Learning Transferable Visual Models From Natural Language Supervision
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models cites Learning Transferable Visual Models From Natural Language Supervision
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection cites Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Dual-Prior Augmented Decoding Network for Long Tail Distribution in HOI Detection cites Learning Transferable Visual Models From Natural Language Supervision
- Dual-Prior Augmented Decoding Network for Long Tail Distribution in HOI Detection cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
Cited by top-tier papers29
- Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation ModelsYichao Cao, Qingfei Tang, Xiu Su, Song Chen et al.NeurIPS 2023 · 64 citations
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI DetectionQinqian Lei, Bo Wang, Robby T. TanNeurIPS 2024 · 42 citations
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion ModelsLiulei Li, Wenguan Wang, Yi YangNeurIPS 2024 · 29 citations
- Open-World Human-Object Interaction Detection via Multi-Modal PromptsJie Yang, Bingliang Li, Ailing Zeng, Lei Zhang et al.CVPR 2024 · 18 citations
- Dual-Prior Augmented Decoding Network for Long Tail Distribution in HOI DetectionJiayi Gao, Kongming Liang, Tao Wei, Wei Chen et al.AAAI 2024 · 14 citations
Builds on31
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
- ViTPose: Simple Vision Transformer Baselines for Human Pose EstimationYufei Xu, Jing Zhang, Qiming Zhang, Dacheng TaoNeurIPS 2022 · 1,105 citations
Related papers
- Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction DetectionYong Zhang, Yingwei Pan, Ting Yao, Rui Huang et al.CVPR 2022 · 88 citations
- End-to-End Human Object Interaction Detection With HOI TransformerCheng Zou, Bohan Wang, Yue Hu, Junqi Liu et al.CVPR 2021
- What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionsA. S. M. Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li et al.CVPR 2022 · 50 citations
- Exploring Predicate Visual Context in Detecting of Human-Object InteractionsFrederic Z. Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong et al.ICCV 2023 · 86 citations
- QPIC: Query-Based Pairwise Human-Object Interaction Detection With Image-Wide Contextual InformationMasato Tamura, Hiroki Ohashi, Tomoaki YoshinagaCVPR 2021
