Lune

ICLR2024Top-tier venue

Consistent Video-to-Video Transfer Using Synthetic Dataset

Jiaxin Cheng, Tianjun Xiao, Tong He

Abstract

We introduce a novel and efficient approach for text-based video-to-video editing that eliminates the need for resource-intensive per-video-per-model finetuning. At the core of our approach is a synthetic paired video dataset tailored for video-tovideo transfer tasks. Inspired by Instruct Pix2Pix's image transfer via editing instruction, we adapt this paradigm to the video domain. Extending the Promptto-Prompt to videos, we efficiently generate paired samples, each with an input video and its edited counterpart. Alongside this, we introduce the Long Video Sampling Correction during sampling, ensuring consistent long videos across batches. Our method surpasses current methods like Tune-A-Video, heralding substantial progress in text-based video-to-video editing and suggesting exciting avenues for further exploration and deployment. https://github.com/ amazon-science/instruct-video-to-video/tree/main Make the car red Porsche and drive alone beach (Editing type: Multiple) Make it watercolor (Editing type: Style) Make it snowy (Editing type: Background) Original video

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

27 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
  • MiniMax-Remover: Taming Bad Noise Helps Video Object Removal cites ReVideo: Remake a Video with Motion and Content Control
  • ReVideo: Remake a Video with Motion and Content Control cites High-Resolution Image Synthesis with Latent Diffusion Models
  • ReVideo: Remake a Video with Motion and Content Control cites Diffusion Models Beat GANs on Image Synthesis
  • ReVideo: Remake a Video with Motion and Content Control cites Learning Transferable Visual Models From Natural Language Supervision
  • ReVideo: Remake a Video with Motion and Content Control cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Implicit Models
  • IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment cites Learning Transferable Visual Models From Natural Language Supervision
  • IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment cites Denoising Diffusion Implicit Models
  • IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment cites Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
  • Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
  • EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning cites Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
  • Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset cites EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
  • Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers27

  1. ReVideo: Remake a Video with Motion and Content ControlChong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang et al.NeurIPS 2024 · 87 citations
  2. Scaling Instruction-Based Video Editing with a High-Quality Synthetic DatasetQingyan Bai, Qiuyu Wang, Hao Ouyang, Yue Yu et al.CVPR 2026 · 79 citations
  3. EditVerse: Unifying Image and Video Editing and Generation with In-Context LearningXuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang et al.ICLR 2026 · 56 citations
  4. MiniMax-Remover: Taming Bad Noise Helps Video Object RemovalBojia Zi, Weixuan Peng, Xianbiao Qi, Jianan Wang et al.NeurIPS 2025 · 43 citations
  5. IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing AssessmentYinan Chen, Jiangning Zhang, Teng Hu, Yuxiang Zeng et al.ICLR 2026 · 29 citations

Builds on39

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  4. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  5. Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations

Related papers

  1. EasyV2V: A High-quality Instruction-based Video Editing FrameworkJinjie Mai, Chaoyang Wang, Gordon Guocheng Qian, Willi Menapace et al.CVPR 2026 · 12 citations
  2. InsViE-1M: Effective Instruction-Based Video Editing with Elaborate Dataset ConstructionYuhui Wu, Liyi Chen, Ruibin Li, Shihao Wang et al.ICCV 2025 · 6 citations
  3. Distilling Vision-Language Models on Millions of VideosYue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu et al.CVPR 2024
  4. Temporal In‑Context Fine‑Tuning with Temporal Reasoning for Versatile Control of Video Diffusion ModelsKinam Kim, Junha Hyung, Jaegul ChooNeurIPS 2025 · 4 citations
  5. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video GeneratorsLevon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel et al.ICCV 2023 · 800 citations
Dusk over the sea between two cliffs drawn in fine vertical lines