Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation

Adversarial training has emerged as a popular approach for training models that are robust to inference-time adversarial attacks. However, our theoretical understanding of why and when it works remains limited. Prior work has offered generalization analysis of adversarial training, but they are either restricted to the Neural Tangent Kernel (NTK) regime or they make restrictive assumptions about data such as (noisy) linear separability or robust realizability. In this work, we study the stability and generalization of adversarial training for two-layer networks without any data distribution assumptions and beyond the NTK regime. Our findings suggest that for networks with any given initialization and sufficiently large width, the generalization bound can be effectively controlled via early stopping. We further improve the generalization bound by leveraging smoothing using Moreau's envelope.

T ) due to non-smoothness. Lei and Ying [2020] tackled the non-smoothness differently by assuming the gradient of the loss to be Hölder continuous. For non-convex and non-smooth loss, Lei [2023] introduced the stability of sub-gradient, as convergence to local minimizers is observed in this setting.

The standard method of giving a generalization guarantee is through uniform convergence. These theories typically yield an upper bound of O( 1 √ n ) and require a large number of training samples in order to get a small generalization gap. Techniques in this category include analyzing the Rademacher complexity [