Lune

AAAI2025Top-tier venue

Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

Jinrong Zhang, Penghui Wang, Chunxiao Liu, Wei Liu, Dian Jin, Qiong Zhang, Erli Meng, Zhengnan Hu

Abstract

To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, Image Prompt Paradigm. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named MI Grounding for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • RegionCLIP: Region-based Language-Image Pretraining cites Learning Transferable Visual Models From Natural Language Supervision
  • GLIPv2: Unifying Localization and Vision-Language Understanding cites RegionCLIP: Region-based Language-Image Pretraining
  • GLIPv2: Unifying Localization and Vision-Language Understanding cites Objects365: A Large-Scale, High-Quality Dataset for Object Detection
  • GLIPv2: Unifying Localization and Vision-Language Understanding cites Learning Transferable Visual Models From Natural Language Supervision
  • PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training cites RegionCLIP: Region-based Language-Image Pretraining
  • PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training cites Objects365: A Large-Scale, High-Quality Dataset for Object Detection
  • PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training cites Learning Transferable Visual Models From Natural Language Supervision
  • PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
  • Breaking the Regional Perception Bottleneck of Multimodal Large Language Models via External Reasoning Framework cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched TrainingWeifu Fu, Jinyang Li, Bin-Bin Gao, Jialin Li et al.CVPR 2026 · 3 citations
  2. Breaking the Regional Perception Bottleneck of Multimodal Large Language Models via External Reasoning FrameworkJinrong Zhang, Zhaoyang Xu, Xusheng He, Xinrui Li et al.CVPR 2026

Builds on10

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  3. Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng et al.ICCV 2019 · 1,018 citations
  4. RegionCLIP: Region-based Language-Image PretrainingYiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li et al.CVPR 2022 · 481 citations
  5. GLIPv2: Unifying Localization and Vision-Language UnderstandingHaotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen et al.NeurIPS 2022 · 403 citations

Related papers

  1. X-SAM: From Segment Anything to Any SegmentationHao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang et al.AAAI 2026 · 16 citations
  2. Prompt-Driven Referring Image Segmentation with Instance ContrastingChao Shang, Zichen Song, Heqian Qiu, Lanxiao Wang et al.CVPR 2024 · 20 citations
  3. Visual in-Context PromptingFeng Li, Qing Jiang, Hao Zhang, Tianhe Ren et al.CVPR 2024
  4. ST-SAM: Multimodal Scene Text Segmentation with Dense Visual and Sparse Textual Prompts via SAMJin Wei, Yaqiang Wu, Jiayi Yan, Zeng Li et al.AAAI 2026
  5. Multimodal Inplace Prompt Tuning for Open-set Object DetectionGuilin Li, Mengdan Zhang, Xiawu Zheng, Peixian Chen et al.ACM MM 2024 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines