Unified In-Context Video Editing

Recent advances in text-to-video generation have sparked interest in generative video editing tasks. Previous methods often rely on task-specific architectures (e.g., additional adapter modules) or dedicated customizations (e.g., DDIM inversion), which limit the integration of versatile editing conditions and the unification of various editing tasks. In this paper, we introduce UNified In-Context Video Editing (UNIC), a simple yet effective framework that unifies diverse video editing tasks within a single model in an in-context manner. To achieve this unification, we represent the inputs of various video editing tasks as three types of tokens: the source video tokens, the noisy video latent, and the multi-modal conditioning tokens that vary according to the specific editing task. Based on this formulation, our key insight is to integrate these three types into a single consecutive token sequence and jointly model them using the native attention operations of DiT, thereby eliminating the need for task-specific adapter designs. Nevertheless, direct task unification under this framework is challenging, leading to severe token collisions and task confusion due to the varying video lengths and diverse condition modalities across tasks. To address these, we introduce task-aware RoPE to facilitate consistent temporal positional encoding, and condition bias that enables the model to clearly differentiate different editing tasks. This allows our approach to adaptively perform different video editing tasks by referring the source video and varying condition tokens "in context", and support flexible task composition. To validate our method, we construct a unified video editing benchmark containing six representative video editing tasks. Results demonstrate that our unified approach achieves superior performance on each task and exhibits emergent task composition abilities. * Equal contribution. Work done during an internship at KwaiVGI, Kuaishou Technology. † Corresponding author. Preprint. Under review. Camera pose Pull out Edited First Frame Style Image A woman is adjusting a dark shiny jacket over her shoulders. A man with short hair is inside a high-tech chamber. ID Ref Image Diverse Video Editing Tasks Emergent Task Composition Source Video Multi-Modal Conditions Generated Results Source Video Multi-Modal Conditions Generated Results ID Insertion Re-camera Control Stylization First-Frame Propagation ID delete ID Swap Source Video Multi-Modal Conditions Generated Results Re-camera Control + Stylization Unified In-Context Video Editing ID Ref Image Source Video Multi-Modal Conditions Generated Results ID insert + Stylization Style Image Style Image A man wearing a tie is walking. Camera pose ARC right A cow with wings is flying over a landscape.