Understanding and Improving Fast Adversarial Training against $l_0$ Bounded Perturbations

This work studies fast adversarial training against sparse adversarial perturbations bounded by l 0 norm. We first demonstrate the unique challenges of employing 1 -step attacks on l 0 bounded perturbations, especially catastrophic overfitting (CO) that cannnot be properly addressed by existing fast adversarial training method for other l p norms ( p ≥ 1 ). We highlight that CO in l 0 adversarial training arises from sub-optimal perturbation locations of 1 -step attack. Some strategies like multi-(cid:15) can mitigate this sub-optimality to some extent, they lead to unstable training in turn. Theoretical and numerical analyses also reveal that the loss landscape of l 0 adversarial training is more craggy than its l ∞ , l 2 and l 1 counterparts, which exaggerates CO. To address this issue, we adopt soft labels and the trade-off loss function to smooth the adversarial loss landscape. Extensive experiments demonstrate our method can overcome the challenge of CO, achieve state-of-the-art performance, and narrow the performance gap between 1 -step and multi-step adversarial training against sparse attacks. Codes are available at https://github.com/CityU-MLO/sPGD.