Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing
Xiaoxue Chen, Tianyu Liu, Hao Zhao, Guyue Zhou, Ya-Qin Zhang
Abstract
Multi-task indoor scene understanding is widely considered as an intriguing formulation, as the affinity of different tasks may lead to improved performance. In this paper, we tackle the new problem of joint semantic, affordance and attribute parsing. However, successfully resolving it requires a model to capture long-range dependency, learn from weakly aligned data and properly balance sub-tasks during training. To this end, we propose an attention-based architecture named Cerberus and a tailored training framework. Our method effectively addresses aforementioned challenges and achieves state-of-the-art performance on all three tasks. Moreover, an in-depth analysis shows concept affinity consistent with human cognition, which inspires us to explore the possibility of weakly supervised learning. Surprisingly, Cerberus achieves strong results using only 0.1% -1% annotation. Visualizations further confirm that this success is credited to common attention maps across tasks. Code and models can be accessed at https: //github.com/OPEN-AIR-SUN/Cerberus .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers10
- Representation Surgery for Multi-Task Model MergingEnneng Yang, Li Shen, Zhenyi Wang, Guibing Guo et al.ICML 2024 · 96 citations
- Zero-1-to-3: Domain-Level Zero-Shot Cognitive Diagnosis via One Batch of Early-Bird Students towards Three Diagnostic ObjectivesWeibo Gao, Qi Liu, Hao Wang, Linan Yue et al.AAAI 2024 · 33 citations
- TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun DistillationPengfei Li, Beiwen Tian, Yongliang Shi, Xiaoxue Chen et al.NeurIPS 2022 · 32 citations
- PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video GenerationMingju Gao, Kaisen Yang, Huan-ang Gao, Bohan Li et al.CVPR 2026 · 3 citations
- PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction ModelMingju Gao, Yike Pan, Huan-ang Gao, Zongzheng Zhang et al.CVPR 2025
Builds on10
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- Vision Transformers for Dense PredictionRené Ranftl, Alexey Bochkovskiy, Vladlen KoltunICCV 2021 · 2,647 citations
- Group-Free 3D Object Detection via TransformersZe Liu, Zheng Zhang, Yue Cao, Han Hu et al.ICCV 2021 · 368 citations
- Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical CommonsenseYixin Chen, Siyuan Huang, Tao Yuan, Yixin Zhu et al.ICCV 2019 · 130 citations
- Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene CompletionPingping Zhang, Wei Liu, Yinjie Lei, Huchuan Lu et al.ICCV 2019 · 79 citations
Related papers
- Going Beyond Multi-Task Dense Prediction with Synergy Embedding ModelsHuimin Huang, Yawen Huang, Lanfen Lin, Ruofeng Tong et al.CVPR 2024
- Multi-Class Part Parsing With Joint Boundary-Semantic AwarenessYifan Zhao, Jia Li, Yu Zhang, Yonghong TianICCV 2019 · 63 citations
- 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point CloudsDaigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng et al.CVPR 2022 · 80 citations
- Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single ImageYinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng et al.CVPR 2020
- Multi-task View Synthesis with Neural Radiance FieldsShuhong Zheng, Zhipeng Bao, Martial Hebert, Yu-Xiong WangICCV 2023 · 7 citations
