The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models

Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction. In this work, we explore to what extent they handle basic linguistic constructions-active-passive voice, coordination, and relative clauses-that even preschool children can typically master. We present BLA, a novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer-based systems, such as CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested models only marginally benefit when fine-tuned or prompted with construction-specific samples. Yet, the generative BLIP2 shows promising trends, especially in an in-context learning setting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models' basic language abilities. Active-Passive voice T: the woman feeds the man. T: the man is fed by the woman. F: the man feeds the woman. F: the woman is fed by the man. Coordination T: the man wears a wetsuit and carries a surfboard. T: the woman wears a red bikini and rides a red bike. F: the man wears a wetsuit and rides a red bike. F: the woman carries a surfboard and wears a red bikini.