CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse barcode DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies. Recently, BioCLIP [61] used CLIP-style contrastive learning [50] to align images with common names and taxonomic descriptions to classify plants, animals, and fungi. While they showed that aligning image representations to text can help improve classification, taxonomic labels, which are not always available to the species level, are needed to obtain text descriptions. In this work, we study whether, by aligning to DNA barcodes (instead of text) during pretraining, we can learn improved representations of images for use in tasks relevant to biodiversity. We propose CLIBD, which uses contrastive learning to map taxonomic labels, biological images and barcode DNA to the same embedding space. By leveraging DNA barcodes, we eliminate the reliance on manual taxonomic labels (as used for BioCLIP) while still incorporating rich taxonomic information into the representation. This is advantageous since DNA barcodes can be obtained at scale more readily than taxonomic labels, which require manual inspection from a human expert [23, 24, 60] . We also investigate leveraging partial taxonomic annotations, when available,