CVPR Findings2026
Extending Segment Anything Model 2 to Multi-Object Tracking by Optimizing Hierarchical Trajectory Memory
Cheng-Yen Yang, Hsiang-Wei Huang, Kuang-Ming Chen, Kunjun Li, Jenq-Neng Hwang
Abstract
The dominant two-stage tracking-by-detection (TbD) and end-to-end tracking-by-query (TbQ) paradigms in Multi-Object Tracking (MOT) present a trade-off between structural management and associative power. While foundation models like SAM2 excel at single-object tracking, adapting them to MOT is challenging. We introduce SAMURAI++, a framework that thoughtfully merges both paradigms to effectively adapt SAM2 for MOT in a training-free setting. Our framework repurposes a TbD pipeline for robust tracklet management, using bipartite matching to resolve the many-to-one matching ambiguities inherent in applying query-based models to MOT. Simultaneously, the querybased model overcomes the TbD performance ceiling by generating supplementary candidates to ensure trajectory continuity when detections are missed. In addition, we propose a hierarchical trajectory memory that consists of short-term motion-aware memory and long-term instance memory, which can effectively refine the trajectory memory with better memory modeling and the concept of bipartite matching under zero-shot settings. The proposed framework bridges the gap between tracking-by-detection and tracking-by-query by offering a unified memory-guided formulation, alleviates the upper-bound limitations imposed by detection quality in TbD pipelines, and enables direct use of single-object tracking models for MOT without requiring training on MOT datasets. Evaluations on DanceTrack and MOT17 demonstrate that SAMURAI++ achieves superior tracking performance with stronger identity preservation and robustness to motion.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MeMOT: Multi-Object Tracking with Memory cites TrackFormer: Multi-Object Tracking with Transformers
- MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking cites TrackFormer: Multi-Object Tracking with Transformers
- MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking cites MeMOT: Multi-Object Tracking with Memory
- MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking cites DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
- DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion cites TrackFormer: Multi-Object Tracking with Transformers
- Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking cites TrackFormer: Multi-Object Tracking with Transformers
- Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking cites DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
Builds on15
- TrackFormer: Multi-Object Tracking with TransformersTim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, Christoph FeichtenhoferCVPR 2022 · 927 citations
- DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse MotionPeize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan et al.CVPR 2022 · 305 citations
- MeMOT: Multi-Object Tracking with MemoryJiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong et al.CVPR 2022 · 216 citations
- Hybrid-SORT: Weak Cues Matter for Online Multi-Object TrackingMingzhan Yang, Guangxin Han, Bin Yan, Wenhua Zhang et al.AAAI 2024 · 171 citations
- MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object TrackingRuopeng Gao, Limin WangICCV 2023 · 143 citations
Related papers
- SAM2MOT: A Novel Paradigm of Multi-Object Tracking by SegmentationJunjie Jiang, Zelin Wang, Manqi Zhao, Yin Li et al.AAAI 2026 · 19 citations
- ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and AssociationShuxiao Ding, Lukas Schneider, Marius Cordts, Juergen GallCVPR 2024
- Dual-Path Temporal Decoder for End-to-End Multi-Object TrackingHyunseop Kim, Juheon Jeong, Hanul Kim, Yeong Jun KohNeurIPS 2025 · 4 citations
- SAM2-OV: A Novel Detection-Only Tuning Paradigm for Open-Vocabulary Multi-Object TrackingYangkai Chen, Qiangqiang Wu, Guangyao Li, Junlong Gao et al.AAAI 2026
- LA-MOTR: End-to-End Multi-Object Tracking by Learnable AssociationPeng Wang, Yongcai Wang, Hualong Cao, Wang Chen et al.ICCV 2025 · 9 citations
