Asymmetric Relational-Geometry Driven Universal Adversarial Perturbations for Vision-Language Models

Although vision-language pre-trained (VLP) models have achieved remarkable success across multimodal tasks, they remain vulnerable to adversarial perturbations. Existing universal adversarial perturbation (UAP) methods in multimodal settings-whether generatorbased or optimization-based-often suffer from limited cross-model transferability, especially in black-box scenarios. We attribute this limitation to the prevalent use of symmetric or distribution-level objectives that overlook the asymmetric roles of image and text modalities and the relational nature of vision-language representations. To address this issue, we propose ARG-Attack, an optimization-based framework that learns universal perturbations under an asymmetric relational-geometry driven objective. Our method integrates three complementary components: a cosine-based loss that induces directional semantic drift in visual features, a center shift loss that geometrically regularizes adversarial embeddings toward a shared semantic center, and a relational polarity loss that explicitly disrupts image-text matching relationships. Together, these objectives enable effective cross-modal interaction without relying on model-specific training losses or probabilistic distribution matching. In addition, we adopt an adaptive gradient update strategy inspired by Adam optimization to stabilize training and accelerate convergence. Extensive experiments across multiple vision-language models and tasks demonstrate that ARG-Attack achieves competitive white-box performance and significantly outperforms state-of-the-art methods in black-box transfer settings.