PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

Figure 1. PARC prompt sensitivity analysis framework overview. Given a collection of VLMs and datasets, PARC identifies which prompt variations these VLMs are most sensitive to, and which VLMs are most agnostic to prompt variations [green]. To achieve this, PARC first applies systematic prompt variations [orange] to the language and vision components of the datasets, then evaluates the VLM performance on these varied datasets with multiple established scores and a novel reliability score [blue], and finally calibrates [red]

those scores to make them directly comparable across the diverse input datasets as well as PARC's prompt variations.