Lune

CVPR2025Top-tier venue

A Distractor-Aware Memory for Visual Object Tracking with SAM2

Jovana Videnovic, Alan Lukezic, Matej Kristan

Abstract

Memory-based trackers are video object segmentation methods that form the target model by concatenating recently tracked frames into a memory buffer and localize the target by attending the current image to the buffered frames. While already achieving top performance on many benchmarks, it was the recent release of SAM2 that placed memory-based trackers into focus of the visual object tracking community. Nevertheless, modern trackers still struggle in the presence of distractors. We argue that a more sophisticated memory model is required, and propose a new distractor-aware memory model for SAM2 and an introspection-based update strategy that jointly addresses the segmentation accuracy as well as tracking robustness. The resulting tracker is denoted as SAM2.1++. We also propose a new distractor-distilled DiDi dataset to study the distractor problem better. SAM2.1++ outperforms SAM2.1 and related SAM memory extensions on seven benchmarks and sets a solid new state-of-the-art on six of them. The code and the new dataset will be available on https: //github.com/jovanavidenovic/DAM4SAM .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

18 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval cites Segment Anything
  • Learning Spatio-Temporal Transformer for Visual Tracking cites Learning Discriminative Model Prediction for Tracking
  • Transforming Model Prediction for Tracking cites Learning Spatio-Temporal Transformer for Visual Tracking
  • Transforming Model Prediction for Tracking cites Learning Discriminative Model Prediction for Tracking
  • ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting cites Segment Anything
  • MixFormer: End-to-End Tracking with Iterative Mixed Attention cites Learning Spatio-Temporal Transformer for Visual Tracking
  • MixFormer: End-to-End Tracking with Iterative Mixed Attention cites Learning Discriminative Model Prediction for Tracking
  • Tracking and Understanding Object Transformations cites Segment Anything
  • Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation cites Segment Anything
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers18

  1. ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian SplattingJiayu Ding, Xinpeng Liu, Zhiyi Pan, Shiqiang Long et al.CVPR 2026 · 7 citations
  2. Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object SegmentationHaichao Jiang, Tianming Liang, Wei-Shi Zheng, Jian-Fang HuCVPR 2026 · 7 citations
  3. Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory RetrievalJing Zhang, Zhikai Li, Xuewen Liu, Qingyi GuICLR 2026 · 5 citations
  4. Tracking and Understanding Object TransformationsYihong Sun, Xinyu Yang, Jennifer J. Sun, Bharath HariharanNeurIPS 2025 · 5 citations
  5. OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex ScenariosHong Gao, Jingyu Wu, Xiangkai Xu, Kangni Xie et al.CVPR 2026 · 4 citations

Builds on23

  1. Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
  2. Learning Discriminative Model Prediction for TrackingGoutam Bhat, Martin Danelljan, Luc Van Gool, Radu TimofteICCV 2019 · 1,294 citations
  3. Learning Spatio-Temporal Transformer for Visual TrackingBin Yan, Houwen Peng, Jianlong Fu, Dong Wang et al.ICCV 2021 · 1,062 citations
  4. MixFormer: End-to-End Tracking with Iterative Mixed AttentionYutao Cui, Cheng Jiang, Limin Wang, Gangshan WuCVPR 2022 · 746 citations
  5. Transforming Model Prediction for TrackingChristoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul et al.CVPR 2022 · 399 citations

Related papers

  1. Memory as Dynamics: Learning Reliability-Guided Predictive Models for Online Video PerceptionMinwoo Kim, Sang Min YoonICML 2026
  2. Robust Ego-Exo Correspondence with Long-Term MemoryYijun Hu, Bing Fan, Xin Gu, Haiqing Ren et al.NeurIPS 2025 · 2 citations
  3. Video-based Person Re-identification with Spatial and Temporal Memory NetworksChanho Eom, Geon Lee, Junghyup Lee, Bumsub HamICCV 2021 · 107 citations
  4. Online Multiple Object Tracking With Cross-Task SynergySong Guo, Jingya Wang, Xinchao Wang, Dacheng TaoCVPR 2021
  5. LOMM: Latest Object Memory Management for Temporally Consistent Video Instance SegmentationSeunghun Lee, Jiwan Seo, Minwoo Choi, Kiljoon Han et al.ICCV 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines