Task-Aware Mechanism: Hybrid MoE Vision Tower Towards Holistic Video Understanding

Should an LVLM use the same video processing pipeline for comprehending the main idea of a 2-hour movie and counting the birds appearing in a 15-second clip? We present Task-Aware Mechanism (TAM), a hybrid-gated Mixture-of-Experts (MoE) vision tower that adapts frame count and resolution to the user query and video length. TAM introduces a compact 0.1B text-only Inductor trained on our TA-116K dataset to infer task types, enabling task-aware visual budgeting and routing: a soft-gated MoE vision encoder for stability, and hard-gated resolution-specific projectors/pipelines for efficient specialization. The explicit budget also separates task classification from downstream visual-language reasoning, making routing decisions easier to inspect and helping diagnose when fixed visual pipelines under-sample temporal or spatial evidence. Built on Qwen2-7B, TallVA-8B-A7B achieves state-of-the-art performance among models with comparable LLMs on most video benchmarks and remains competitive against stronger-LLM baselines, showing that task-aware visual budgeting makes video understanding more holistic.