Transformative or Conservative? Conservation laws for ResNets and Transformers
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Abstract
While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practical architectures. This paper bridges this gap by deriving and analyzing conservation laws for modern architectures, with a focus on convolutional ResNets and Transformer networks. For this, we first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws, and no more than the known ones. In the case of a single attention layer, we also completely describe all conservation laws, and we show that residual blocks have the same conservation laws as the same block without skip connection. We then introduce the notion of conservation laws that depend only on a subset of parameters (corresponding e.g. to a pair of consecutive layers, to a residual block, or to an attention layer). We demonstrate that the characterization of such laws can be reduced to the analysis of the corresponding building block in isolation. Finally, we examine how these newly discovered conservation principles, initially established in the continuous gradient flow regime, persist under discrete optimization dynamics, particularly in the context of Stochastic Gradient Descent (SGD).
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Never Saddle for Reparameterized Steepest Descent as Mirror Flow cites Abide by the law and follow the flow: conservation laws for gradient flows
- Conservation Laws for Modern Neural Architectures cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Conservation Laws for Modern Neural Architectures cites Abide by the law and follow the flow: conservation laws for gradient flows
- Conservation Laws for Modern Neural Architectures cites Understanding the Dynamics of Gradient Flow in Overparameterized Linear models
- Conservation Laws for Modern Neural Architectures cites On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks
- Intrinsic training dynamics of deep neural networks cites Abide by the law and follow the flow: conservation laws for gradient flows
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- Abide by the law and follow the flow: conservation laws for gradient flows cites Understanding the Dynamics of Gradient Flow in Overparameterized Linear models
- Abide by the law and follow the flow: conservation laws for gradient flows cites On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks
Cited by top-tier papers3
- Intrinsic training dynamics of deep neural networksSibylle Marcotte, Gabriel Peyré, Rémi GribonvalICLR 2026 · 4 citations
- Never Saddle for Reparameterized Steepest Descent as Mirror FlowTom Jacobs, Chao Zhou, Rebekka BurkholzICLR 2026 · 3 citations
- Conservation Laws for Modern Neural ArchitecturesViet Hoang Tran, VINH KHANH BUI, Ngoc Tan Lai, Nam Nguyen et al.ICML 2026
Builds on7
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Understanding the Dynamics of Gradient Flow in Overparameterized Linear modelsSalma Tarmoun, Guilherme França, Benjamin D. Haeffele, René VidalICML 2021 · 76 citations
- Abide by the law and follow the flow: conservation laws for gradient flowsSibylle Marcotte, Rémi Gribonval, Gabriel PeyréNeurIPS 2023 · 54 citations
- On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear NetworksHancheng Min, Salma Tarmoun, René Vidal, Enrique MalladaICML 2021 · 53 citations
Related papers
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning DynamicsDaniel Kunin, Javier Sagastuy-Breña, Surya Ganguli, Daniel L. K. Yamins et al.ICLR 2021 · 100 citations
- Keep the Momentum: Conservation Laws beyond Euclidean Gradient FlowsSibylle Marcotte, Rémi Gribonval, Gabriel PeyréICML 2024 · 8 citations
- Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow AnalysisHongru Yang, Bhavya Kailkhura, Zhangyang Wang, Yingbin LiangNeurIPS 2024 · 14 citations
- Are GATs Out of Balance?Nimrah Mustafa, Aleksandar Bojchevski, Rebekka BurkholzNeurIPS 2023 · 11 citations
- A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From DepthYiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu et al.ICML 2020 · 85 citations
