EthoCLIP: Ontology-Enhanced Video-Language Pretraining for Animal Behavior Understanding

Vision-language models (VLMs) have achieved remarkable success across numerous domains, yet they lag significantly in animal behavior understanding due to severe data scarcity. Annotated animal behavior videos are prohibitively expensive and time-consuming to collect, requiring domain expertise and controlled observation conditions. To address this challenge, we leverage structured domain knowledge as an inductive bias from the Neuro Behavior Ontology (NBO), which provides professional annotations, hierarchical behavior structures, and comprehensive semantic coverage. We construct Animal-Band, an NBO-consistent dataset integrating 74,671 videos across multiple species and behaviors with semantic standardization and extended knowledge. Based on this resource, we present EthoCLIP, an ontology-enhanced vision-language contrastive learning framework that embeds ontology semantics through an ontology-aware graph module to capture hierarchical relationships among behaviors and learn structured semantic dependencies. Incorporating ontological information reduces reliance on purely datadriven learning, thereby alleviating needs for large-scale datasets. Extensive experiments validate both our dataset and method. Results demonstrate that EthoCLIP pretrained on AnimalBand substantially improves behavior recognition accuracy and transfer learning performance across diverse benchmarks, confirming that ontology-driven semantic enrichment effectively mitigates data scarcity in animal behavior understanding. Our data and code will be released at https://github.com/PRIS-CV /Ani malBand.