Video Token Merging for Long Video Understanding
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, Xinyu Li
Abstract
As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Efficiently Modeling Long Sequences with Structured State Spaces cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- Is Space-Time Attention All You Need for Video Understanding? cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Is Space-Time Attention All You Need for Video Understanding? cites Language Models are Few-Shot Learners
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Cited by top-tier papers6
- Mitigating Semantic Collapse in Partially Relevant Video RetrievalWonJun Moon, Minseok Jung, Gilhan Park, Tae-Young Kim et al.NeurIPS 2025 · 7 citations
- FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video UnderstandingJanghoon Cho, Jungsoo Lee, Munawar Hayat, Kyuwoong Hwang et al.ICLR 2026 · 6 citations
- Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language ModelsJialiang Zhang, Junlong Tong, Junyan Lin, Hao Wu et al.CVPR 2026 · 6 citations
- SURGE: Surprise-Guided Token Reduction for Efficient Video Understanding with VLMsChong Tang, Sannara Ek, Dirk Koch, Robert Mullins et al.ICLR 2026
- UniAPO: Unified Multimodal Automated Prompt OptimizationQipeng Zhu, Yanzhe Chen, Huasong Zhong, Jie Chen et al.AAAI 2026
Builds on26
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
- Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius, Heng Wang, Lorenzo TorresaniICML 2021 · 2,927 citations
Related papers
- vid-TLDR: Training Free Token merging for Light-Weight Video TransformerJoonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi et al.CVPR 2024
- Selective Structured State-Spaces for Long-Form Video UnderstandingJue Wang, Wentao Zhu, Pichao Wang, Xiang Yu et al.CVPR 2023
- Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMsJeongseok Hyun, Sukjun Hwang, Su Ho Han, Taeoh Kim et al.ICCV 2025 · 2 citations
- MeToM: Metadata-Guided Token Merging for Efficient Video LLMsZhuojie Wu, Shijie Wang, Xin YuCVPR 2026
- Token Merging: Your ViT But FasterDaniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang et al.ICLR 2023 · 62 citations
