Decoupling Dense Video Captioning via Task-specific Prompts

Dense video captioning aims to generate descriptive sentences for each temporally localized event in a video. This task comprises two subtasks: event detection and event captioning. Existing methods commonly adopt a DETR-like (Detection Transformer) architecture to perform both subtasks in parallel. These methods assume that both subtasks require the same visual information and thus extract a single event representation for each event using a shared query. We observe that event detection and event captioning emphasize different regions of a video. In particular, compared to event captioning, event detection tends to focus more on the boundary regions of event proposals. Therefore, relying on shared queries may hinder the ability of the model to meet the specific needs of each subtask, leading to suboptimal performance. In this paper, we propose decoupling the two subtasks by assigning distinct queries to each, enabling more accurate capture of task-specific features. Specifically, we introduce a task-specific query transformation module. This module utilizes two sets of task-specific prompts to transform shared queries into queries tailored for each subtask. These task-specific queries enable each subtask to attend to the video regions that are most beneficial to its respective objectives. By integrating our method into several state-of-the-art frameworks, we achieve superior performance on both event detection and event captioning.