Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

Antoine Yang; Arsha Nagrani; Paul Hongsuck Seo; Antoine Miech; Jordi Pont-Tuset; Ivan Laptev; Josef Sivic; Cordelia Schmid

doi:10.1109/cvpr52729.2023.01032

CVPR2023顶会

Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

arXiv DOI 出版方

2023年份

123顶会引用

摘要

Dense video captioning Hey guys today I am going to teach you how to ski The kids make it look easy First slope, congratz! Vid2Seq <1s><8s>The man is fastening the dog. <20s><50s>The dogs are pulling the sled. <45s><49s>The man is saying hello.

Figure 1. Vid2Seq is a visual language model that predicts dense event captions together with their temporal grounding in the video by generating a single sequence of tokens (right). This ability is enabled by large-scale pretraining on unlabeled narrated videos (left).