TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation
Pengfei Li, Beiwen Tian, Yongliang Shi, Xiaoxue Chen, Hao Zhao, Guyue Zhou, Ya-Qin Zhang
Abstract
Current referring expression comprehension algorithms can effectively detect or segment objects indicated by nouns, but how to understand verb reference is still under-explored. As such, we study the challenging problem of task oriented detection, which aims to find objects that best afford an action indicated by verbs like sit comfortably on. Towards a finer localization that better serves downstream applications like robot interaction, we extend the problem into task oriented instance segmentation. A unique requirement of this task is to select preferred candidates among possible alternatives. Thus we resort to the transformer architecture which naturally models pair-wise query relationships with attention, leading to the TOIST method. In order to leverage pre-trained noun referring expression comprehension models and the fact that we can access privileged noun ground truth during training, a novel noun-pronoun distillation framework is proposed. Noun prototypes are generated in an unsupervised manner and contextual pronoun features are trained to select prototypes. As such, the network remains noun-agnostic during inference. We evaluate TOIST on the large-scale task oriented dataset COCO-Tasks and achieve +10.9% higher than the best-reported results. The proposed noun-pronoun distillation can boost and by +2.8% and +3.8%. Codes and models are publicly available at https://github.com/AIR-DISCOVER/TOIST.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers6
- RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression SegmentationChangli Wu, Qi Chen, Jiayi Ji, Haowei Wang et al.NeurIPS 2024 · 16 citations
- LottieGPT: Tokenizing Vector Animation for Autoregressive GenerationJunhao Chen, Kejun Gao, Yuehan Cui, Mingze Sun et al.CVPR 2026 · 10 citations
- RESAnything: Attribute Prompting for Arbitrary Referring SegmentationRuiqi Wang, Hao ZhangNeurIPS 2025 · 6 citations
- Butter: Frequency Consistency and Hierarchical Fusion for Autonomous Driving Object DetectionXiaojian Lin, Wenxin Zhang, Yuchu Jiang, Wangyu Wu et al.ACM MM 2025 · 3 citations
- PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction ModelMingju Gao, Yike Pan, Huan-ang Gao, Zongzheng Zhang et al.CVPR 2025
Builds on12
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
- Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
- VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
Related papers
- SOIT: Segmenting Objects with Instance-Aware TransformersXiaodong Yu, Dahu Shi, Xing Wei, Ye Ren et al.AAAI 2022 · 32 citations
- CAKE: Category Aware Knowledge Extraction for Open-Vocabulary Object DetectionShiyuan Ma, Donglin Qian, Kai Ye, Shengchuan ZhangAAAI 2025 · 8 citations
- SUGAR : Pre-training 3D Visual Representations for RoboticsShizhe Chen, Ricardo Garcia, Ivan Laptev, Cordelia SchmidCVPR 2024
- Alleviating Catastrophic Forgetting of Incremental Object Detection via Within-Class and Between-Class Knowledge DistillationMengxue Kang, Jinpeng Zhang, Jinming Zhang, Xiashuang Wang et al.ICCV 2023 · 23 citations
- DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiesJiahao Chang, Shuo Wang, Hai-Ming Xu, Zehui Chen et al.ICCV 2023 · 53 citations
