Weight-covariance alignment for adversarially robust neural networks

Stochastic Neural Networks (SNNs) that inject noise into their hidden layers have recently been shown to achieve strong robustness against adversarial attacks. However, existing SNNs are usually heuristically motivated, and often rely on adversarial training, which is computationally costly. We propose a new SNN that achieves state-of-the-art performance without relying on adversarial training, and enjoys solid theoretical justification. Specifically, while existing SNNs inject learned or hand-tuned isotropic noise, our SNN learns an anisotropic noise distribution to op-timize a learning-theoretic bound on adversarial robustness. We evaluate our method on a number of popular benchmarks, show that it can be applied to different architectures, and that it provides robustness to a variety of white-box and black-box attacks, while being simple and fast to train compared to existing alternatives. noise. We address the aforementioned limitations and propose an SNN that makes use of learnable anisotropic noise. We theoretically analyse the margin between the clean and adversarial performance of a stochastic model and derive an upper bound on the difference between these two quantities. This novel theoretical insight suggests that the anisotropic noise covariance in an SNN should be optimized to align with the classifier weights, which has the effect of tight-ening the bound between clean and adversarial performance. This leads to an easy-to-implement regularizer, which can be efficiently optimized on clean samples alone without need for adversarial training. We show that our method, called Weight-Covariance Alignment (WCA), can be applied to architectures of varied depth and complexity (namely, LeNet++ and ResNet-18), and achieves state-of-the-art robustness across several widely used benchmarks, including CIFAR-10, CIFAR-100, SVHN and F-MNIST. Moreover, this high level of robustness is demonstrated for