VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

Many decision-making tasks, where both accuracy and efficiency matter, still require human supervision. For example, tasks like traffic officers reviewing hour-long dashcam footage or researchers screening conference videos can benefit from concise summaries that reduce cognitive load and save time. Yet current vision-language models (VLMs) often produce verbose, redundant outputs that hinder task performance. Existing video caption evaluation depends on costly human annotations and overlooks the summaries' utility in downstream tasks. We address these gaps with Video-to-text Information Bottleneck Evaluation (VIBE), an annotation-free method that scores VLM outputs using two metrics: grounding (how well the summary aligns with visual content) and utility (how informative it is for the task). VIBE selects from randomly sampled VLM outputs by ranking them according to the two scores to support effective human decision-making. Human studies on LearningPaper24, SUTD-TrafficQA, and LongVideoBench show that summaries selected by VIBE consistently improve performance-boosting task accuracy by up to 61.23% and reducing response time by 75.77% compared to naive VLM summaries or raw video. 2

• Equal contribution (Order determined by coin toss). 2 Project Website, Code, and LearningPaper24 Dataset. 39th Conference on Neural Information Processing Systems (NeurIPS 2025).

Video (𝑽) VLM Captioning Task (𝒀) Summaries Candidates (𝑻) "Annotation-Free Rejection Sampling" "Raw Video" This video teaches the physics principle regarding birds conserving body heat due to blood vessels … It references a webpage called "Physics of the Everyday" for learning physics of everyday activities.

The video explores the unique physical adaptations of flamingos, focusing on their ability to stand on one leg and conserve body heat … It features visuals of flamingos throughout the video.

This video from SciShow covers the unique one-legged sleeping stance of flamingos and compares it to a pigeon's roosting behavior, using diagrams and footage to explain their anatomy. It concludes with a promotion for Brilliant's Physics courses.