Lune

RTSS2025Top-tier venue

CF-DETR: Coarse-to-Fine Transformer for Real-Time Object Detection

Woojin Shin, Donghwa Kang, Byeongyun Park, Brent ByungHoon Kang, Jinkyu Lee, Hyeongboo Baek

Abstract

Detection Transformers (DETR) are increasingly adopted in autonomous vehicle (AV) perception systems due to their superior accuracy over convolutional networks. However, concurrently executing multiple DETR tasks presents significant challenges in meeting firm real-time deadlines (R1) and high accuracy requirements (R2), particularly for safety-critical objects, while navigating the inherent latency-accuracy trade-off under resource constraints. Existing real-time DNN scheduling approaches often treat models generically, failing to leverage Transformer-specific properties for efficient resource allocation. To address these challenges, we propose CF-DETR, an integrated system featuring a novel coarse-to-fine Transformer architecture and a dedicated real-time scheduling framework NPFP * * . CF-DETR employs three key strategies (A1: coarse-to-fine inference, A2: selective fine inference, A3: multi-level batch inference) that exploit Transformer properties to dynamically adjust patch granularity and attention scope based on object criticality, aiming to satisfy R2. The NPFP * * scheduling framework (A4) orchestrates these adaptive mechanisms A1-A3. It partitions each DETR task into a safety-critical coarse subtask for guaranteed critical object detection within its deadline (ensuring R1), and an optional fine subtask for enhanced overall accuracy (R2), while managing individual and batched execution. Our extensive evaluations on server, GPU-enabled embedded platforms, and actual AV platforms demonstrate that CF-DETR, under an NPFP * * policy, successfully meets strict timing guarantees for critical operations and achieves significantly higher overall and critical object detection accuracy compared to existing baselines across diverse AV workloads.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
  • A-ViT: Adaptive Tokens for Efficient Vision Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • DETRs Beat YOLOs on Real-time Object Detection cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  3. DETRs Beat YOLOs on Real-time Object DetectionYian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei et al.CVPR 2024 · 3,046 citations
  4. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object DetectionHao Zhang, Feng Li, Shilong Liu, Lei Zhang et al.ICLR 2023 · 753 citations
  5. A-ViT: Adaptive Tokens for Efficient Vision TransformerHongxu Yin, Arash Vahdat, José M. Álvarez, Arun Mallya et al.CVPR 2022 · 288 citations

Related papers

  1. Prophet: Realizing a Predictable Real-time Perception Pipeline for Autonomous VehiclesLiangkai Liu, Zheng Dong, Yanzhi Wang, Weisong ShiRTSS 2022 · 34 citations
  2. RTInfer: Real-Time Inference of Multiple DNNs on Edge GPUsRenjie Li, Tong Sun, Yi Gao, Wei DongICML 2026
  3. CF-DETR: Coarse-to-Fine Transformers for End-to-End Object DetectionXipeng Cao, Peng Yuan, Bailan Feng, Kun NiuAAAI 2022 · 59 citations
  4. LaLaRAND: Flexible Layer-by-Layer CPU/GPU Scheduling for Real-Time DNN TasksWoosung Kang, Kilho Lee, Jinkyu Lee, Insik Shin et al.RTSS 2021 · 68 citations
  5. Real-Time Multitasking of Deep Neural Networks With Nvidia TensorrtFederico Aromolo, Andrea Stevanato, Alessandro Biondi, Giorgio C. ButtazzoRTSS 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines