Lune

ICLR2025Top-tier venue

CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification

Mingkun Zhang, Keping Bi, Wei Chen, Jiafeng Guo, Xueqi Cheng

Abstract

In this paper, we aim to build an adversarially robust zero-shot image classifier. We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts "a photo of a <class-name>.". Purification is the path we choose since it does not require adversarial training on specific attack types and thus can cope with any foreseen attacks. We then formulate purification risk as the KL divergence between the joint distributions of the purification process of denoising the adversarial samples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire us to explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and "a photo of a.". As far as we know, CLIPure is the first purification method in multi-modal latent space and CLIPure-Cos is the first purification method that is not based on generative models, which substantially improves defense efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13 datasets that previous CLIP-based defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on ImageNet, and 108% relative improvements of average robustness on the 13 datasets over previous SOTA. The code is available at https://github.com/TMLResearchGroup-CAS/CLIPure.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • You Don't Protect if You Don't Expect: Breaking the Key Assumption behind CLIP's Test-Time Defenses cites Learning Transferable Visual Models From Natural Language Supervision
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP cites Learning Transferable Visual Models From Natural Language Supervision
  • Diffusion Models Beat GANs on Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Calibrating Uncertainty for Zero-Shot Adversarial CLIP cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Diversifying Counterattacks: Orthogonal Exploration for Robust CLlP Inference cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. You Don't Protect if You Don't Expect: Breaking the Key Assumption behind CLIP's Test-Time DefensesRuize Zhang, Yu Li, Zhang Wan, Juan Cao et al.ICML 2026
  2. Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPSen Nie, Jie Zhang, Zhuo Wang, Shiguang Shan et al.ICML 2026
  3. Vulnerability-Aware Robust Multimodal Adversarial TrainingJunrui Zhang, Xinyu Zhao, Jie Peng, Chenjie Wang et al.AAAI 2026
  4. Calibrating Uncertainty for Zero-Shot Adversarial CLIPWenjing Lu, Zerui Tao, Yuning Qiu, Dongping Zhang et al.ICML 2026
  5. Diversifying Counterattacks: Orthogonal Exploration for Robust CLlP InferenceChengze Jiang, Minjing Dong, Xinli Shi, Jie GuiAAAI 2026

Builds on28

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  5. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations

Related papers

  1. Language-Driven Anchors for Zero-Shot Adversarial RobustnessXiao Li, Wei Zhang, Yining Liu, Zhanhao Hu et al.CVPR 2024
  2. CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPSonglong Xing, Zhengyu Zhao, Nicu SebeCVPR 2025
  3. TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language ModelsXin Wang, Kai Chen, Jiaming Zhang, Jingjing Chen et al.CVPR 2025
  4. Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language ModelsLu Yu, Haiyang Zhang, Changsheng XuNeurIPS 2024 · 29 citations
  5. On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free ApproachBaoshun Tong, Hanjiang Lai, Yan Pan, Jian YinCVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines