Lune

Abstract

Graph neural networks (GNNs) with a rescale invariance, such as GATs, can be re-parameterized during optimization through dynamic rescaling of network parameters and gradients while keeping the loss invariant. In this work, we explore dynamic rescaling as a tool to influence GNN training dynamics in two key ways: i) balancing the network with respect to various criteria, and ii) controlling the relative learning speeds of different layers. We gain novel insights, unique to GNNs, that reveal distinct training modes for different tasks. For heterophilic graphs, achieving balance based on relative gradients leads to faster training and better generalization. In contrast, homophilic graphs benefit from delaying the learning of later layers. Additionally, we show that training in balance supports larger learning rates, which can improve generalization. Moreover, controlling layer-wise training speeds is linked to grokking-like phenomena, which may be of independent interest. Deep neural networks (DNNs) with positively homogeneous non-linear activation functions such as ReLU exhibit the rescale invariance property [31], i.e. scaling down incoming weights to a neuron and scaling up the outgoing weights from the neuron by the same factor does not alter the function represented by the network.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Fixed Aggregation Features Can Rival GNNs cites How Attentive are Graph Attention Networks?
  • How Attentive are Graph Attention Networks? cites DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
  • How Attentive are Graph Attention Networks? cites PairNorm: Tackling Oversmoothing in GNNs
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Path-conditioned training: a principled way to rescale ReLU neural networksArthur Lebeurrier, Titouan Vayer, Rémi GribonvalICML 2026 · 3 citations
  2. Fixed Aggregation Features Can Rival GNNsCelia Rubio-Madrigal, Rebekka BurkholzICML 2026 · 2 citations
  3. Grokking Beyond the Euclidean Norm of Model ParametersPascal Tikeng Notsawo Jr., Guillaume Dumas, Guillaume RabusseauICML 2025

Builds on28

  1. How Attentive are Graph Attention Networks?Shaked Brody, Uri Alon, Eran YahavICLR 2022 · 1,717 citations
  2. DropEdge: Towards Deep Graph Convolutional Networks on Node ClassificationYu Rong, Wenbing Huang, Tingyang Xu, Junzhou HuangICLR 2020 · 1,599 citations
  3. Pruning neural networks without any data by iteratively conserving synaptic flowHidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, Surya GanguliNeurIPS 2020 · 884 citations
  4. PairNorm: Tackling Oversmoothing in GNNsLingxiao Zhao, Leman AkogluICLR 2020 · 590 citations
  5. Towards Understanding Grokking: An Effective Theory of Representation LearningZiming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud et al.NeurIPS 2022 · 299 citations

Related papers

  1. Are GATs Out of Balance?Nimrah Mustafa, Aleksandar Bojchevski, Rebekka BurkholzNeurIPS 2023 · 11 citations
  2. From Trainable Negative Depth to Edge Heterophily in GraphsYuchen Yan, Yuzhong Chen, Huiyuan Chen, Minghua Xu et al.NeurIPS 2023 · 41 citations
  3. Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More DepthKeyulu Xu, Mozhi Zhang, Stefanie Jegelka, Kenji KawaguchiICML 2021 · 87 citations
  4. Making Classic GNNs Strong Baselines Across Varying Homophily: A Smoothness-Generalization PerspectiveMing Gu, Zhuonan Zheng, Sheng Zhou, Meihan Liu et al.NeurIPS 2025 · 4 citations
  5. Mixture of Scope Experts at Test: Generalizing Deeper Graph Neural Networks with Shallow VariantsGangda Deng, Hongkuan Zhou, Rajgopal Kannan, Viktor PrasannaNeurIPS 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines