Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel

In this paper, we investigate deep feedforward neural networks with random weights. The input data matrix X is drawn from a Gaussian mixture model. We demonstrate that certain eigenvalues of the conjugate kernel and neural tangent kernel may lie outside the support of their limiting spectral measures in the high-dimensional regime. The existence and asymptotic positions of such isolated eigenvalues are rigorously analyzed. Furthermore, we provide a precise characterization of the entrywise limit of the projection matrix onto the eigenspace associated with these isolated eigenvalues. Our findings reveal that the eigenspace captures inherent group features present in X. This study offers a quantitative analysis of how group features from the input data evolve through hidden layers in randomly weighted neural networks.