Adversarial Patch EXterminator: Zero-Shot and Patch-Agnostic Defense Framework Against Adversarial Patch Attacks

Adversarial patch attacks pose a serious threat to modern computer vision systems. Although existing defense solutions attempt to mitigate such attacks by developing certifiable models or patch identification pipelines, they generally rely on prior knowledge or extensive training data, show insufficient robustness across varying physical conditions, and present limited performance against challenging cases (e.g., tiny, irregular, or highly background-coherent patches). To address such limitations, we propose APEX, a zero-shot, patch-agnostic three-stage adversarial patch defense framework. Specifically, APEX first concentrates patch regions through bounding-box extraction, then integrates a mutual information-based blur heatmap with an edge-aware boundary heatmap to locate adversarial regions, and finally leverages structure-guided image inpainting to restore the image. Our experiments on multiple datasets and existing state-of-the-art defense methods demonstrate that APEX can effectively defend against various types of adversarial patches (e.g., non-naturalistic, naturalistic, and infrared images). In addition, APEX shows superior capability in patch localization, maintains high robustness against varying environments (e.g., lighting conditions) and extreme cases, and also demonstrates high performance in protecting various models in physical-world scenarios.