Unaligned UAV RGBT Tracking: A Largescale Benchmark and a Novel Approach

With the rapid development of the low-altitude economy, multi-modal visual tracking in UAV scenarios has attracted extensive attention. UAVs are typically equipped with independent visible (RGB) and thermal infrared (TIR) sensors, resulting in an inherent spatial misalignment between the two modalities. However, existing RGBT tracking methods generally rely on spatially aligned data inputs, making them unsuitable for unaligned RGBT tracking task in UAV scenarios. In this work, we introduce a new task called unaligned UAV RGBT tracking and construct the first largescale unaligned RGB and TIR video dataset to promote the research and development in this field. The dataset contains 1,453 pairs of UAV-captured RGBT sequences with precise dual-modal bounding box annotations, and covers 42 object categories, 22 typical challenge attributes, and diverse spatial misalignment scales to simulate real-world challenging scenarios better. To address the limitations of existing methods that fail to handle the spatial misalignment issue in UAV scenarios, we propose the novel RGBT tracking approach. In particular, we design a mixture of shift estimation experts module to adaptively estimate the spatial shifts across two modalities at different scales, along with a cross-modal alignment and fusion module to correct feature shifts, compensate for nonlinear deformations, and integrate multi-modal information. Extensive experiments on the created dataset demonstrate that the proposed tracker significantly outperforms existing state-of-theart tracking methods, validating its practicality and robustness in real-world unaligned UAV tracking scenarios.