Non-Vacuous Generalisation Bounds for Shallow Neural Networks

We focus on a specific class of shallow neural networks with a single hidden layer, namely those with L 2 -normalised data and either a sigmoidshaped Gaussian error function ("erf") activation or a Gaussian Error Linear Unit (GELU) activation. For these networks, we derive new generalisation bounds through the PAC-Bayesian theory; unlike most existing such bounds they apply to neural networks with deterministic rather than randomised parameters. Our bounds are empirically non-vacuous when the network is trained with vanilla stochastic gradient descent on MNIST, Fashion-MNIST, and binary classification versions of the above.