On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model
Peizhong Ju, Xiaojun Lin, Ness B. Shroff
摘要
In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式，引用它的顶会 Paper 也一样。你提问，回答直接引用原文。
引用它的顶会 Paper3
- Theory on Forgetting and Generalization of Continual LearningSen Lin, Peizhong Ju, Yingbin Liang, Ness B. ShroffICML 2023 · 被引用 74 次
- HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMsXinyue Zeng, Junhong Lin, Yujun Yan, Feng Guo 等ICLR 2026 · 被引用 13 次
- Theoretical Characterization of the Generalization Performance of Overfitted Meta-LearningPeizhong Ju, Yingbin Liang, Ness B. ShroffICLR 2023 · 被引用 3 次
它引用的顶会 Paper6
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networksZiwei Ji, Matus TelgarskyICLR 2020 · 被引用 193 次
- Double Trouble in Double Descent: Bias and Variance(s) in the Lazy RegimeStéphane d'Ascoli, Maria Refinetti, Giulio Biroli, Florent KrzakalaICML 2020 · 被引用 163 次
- On the Similarity between the Laplace and Neural Tangent KernelsAmnon Geifman, Abhay Kumar Yadav, Yoni Kasten, Meirav Galun 等NeurIPS 2020 · 被引用 118 次
- Overfitting Can Be Harmless for Basis Pursuit, But Only to a DegreePeizhong Ju, Xiaojun Lin, Jia LiuNeurIPS 2020 · 被引用 20 次
- On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel ModelsPeizhong Ju, Xiaojun Lin, Ness B. ShroffICML 2021 · 被引用 13 次
相关 Paper
- What can linearized neural networks actually say about generalization?Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, Pascal FrossardNeurIPS 2021 · 被引用 62 次
- Benign Overfitting in Deep Neural Networks under Lazy TrainingZhenyu Zhu, Fanghui Liu, Grigorios Chrysos, Francesco Locatello 等ICML 2023 · 被引用 12 次
- A Non-Parametric Regression Viewpoint : Generalization of Overparametrized Deep RELU Network Under Noisy ObservationsNamjoon Suh, Hyunouk Ko, Xiaoming HuoICLR 2022 · 被引用 15 次
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural NetworksZixiang Chen, Yuan Cao, Quanquan Gu, Tong ZhangNeurIPS 2020 · 被引用 82 次
- Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural NetworksYu Bai, Jason D. LeeICLR 2020 · 被引用 128 次
