Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation

https://vdm-evfi.github.io/ (a) Frame-only Video Frame Interpolation (b) Pre-trained Video Diffusion Model (c) Event-based Video Frame Interpolation (d) Ours Reference Figure 1. We compare our proposed approach RE-VDM, which adapts a pre-trained video diffusion model for event-based video frame interpolation on unseen real-world data, with three baselines: frame-only interpolation method GIMM-VFI-R-P [14], test-time optimization of a pre-trained video diffusion model via Time Reversal [11], and event-based interpolation method CBMNet-Large [21], trained on the same dataset as our method. The left-most column shows the start frame, end frame, and reference interpolation frames overlaid with events. Leveraging data priors in the pre-trained video diffusion model and the diffusion process, along with motion guidance controlled by input events, our approach demonstrates superior generalization performance on unseen real-world frames with substantial motion.