Neural Tangent Kernels Under Stochastic Data Augmentation

The Neural Tangent Kernel NTK of Equivariant NNs Concrete Examples 4 Data Augmentation vs GCNNs What can we say about the dynamics of NNs without specifying their initial parameters? Let's learn sin(x) -3 -2 -1 0 1 2 3 -1 -0.5 0 0.5 1 training samples 3-layer MLPs with width 8 What can we say about the dynamics of NNs without specifying their initial parameters? Let's learn sin(x) 5 0 0.5 1 training samples 3-layer MLPs with width 64 What can we say about the dynamics of NNs without specifying their initial parameters? Let's learn sin(x) -3 -2 -1 0 1 2 3 -1 -0.5 0 0.5 1 training samples 3-layer MLPs with width 256 What can we say about the dynamics of NNs without specifying their initial parameters? Let's learn sin(x) -3 -2 -1 0 1 2 3 -1 -0.5 0 0.5 1 training samples 3-layer MLPs with width 1024 What can we say about the dynamics of NNs without specifying their initial parameters? Let's learn sin(x) -3 -2 -1 0 1 2 3 -1 -0.5 0 0.5 1 training samples 3-layer MLPs with width 4096 How is the NTK computed in practice? → layer by layer! (Jacot, Gabriel, and Hongler 2018) Convenient: Neural Network Gaussian Process Kernel (NNGP) width limit in practice: Histological image classification 9 classes of microscopical tissue images (Kather, Halama, and Marx 2018