Graph Neural Networks with Learnable Structural and Positional Representations
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, Xavier Bresson
Abstract
Graph neural networks (GNNs) have become the standard learning architectures for graphs. GNNs have been applied to numerous domains ranging from quantum chemistry, recommender systems to knowledge graphs and natural language processing. A major issue with arbitrary graphs is the absence of canonical positional information of nodes, which decreases the representation power of GNNs to distinguish e.g. isomorphic nodes and other graph symmetries. An approach to tackle this issue is to introduce Positional Encoding (PE) of nodes, and inject it into the input layer, like in Transformers. Possible graph PE are Laplacian eigenvectors. In this work, we propose to decouple structural and positional representations to make easy for the network to learn these two essential properties. We introduce a novel generic architecture which we call LSPE (Learnable Structural and Positional Encodings). We investigate several sparse and fully-connected (Transformer-like) GNNs, and observe a performance increase for molecular datasets, from 1.79% up to 64.14% when considering learnable PE for both GNN classes. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
165 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Recipe for a General, Powerful, Scalable Graph Transformer cites Structure-Aware Transformer for Graph Representation Learning
- Recipe for a General, Powerful, Scalable Graph Transformer cites Rethinking Graph Transformers with Spectral Attention
- Recipe for a General, Powerful, Scalable Graph Transformer cites Principal Neighbourhood Aggregation for Graph Nets
- Open Graph Benchmark: Datasets for Machine Learning on Graphs cites DeepGCNs: Can GCNs Go As Deep As CNNs?
- Structure-Aware Transformer for Graph Representation Learning cites DeepGCNs: Can GCNs Go As Deep As CNNs?
- Structure-Aware Transformer for Graph Representation Learning cites Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Structure-Aware Transformer for Graph Representation Learning cites Rethinking Graph Transformers with Spectral Attention
- Structure-Aware Transformer for Graph Representation Learning cites Principal Neighbourhood Aggregation for Graph Nets
- Rethinking Graph Transformers with Spectral Attention cites Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Rethinking Graph Transformers with Spectral Attention cites Principal Neighbourhood Aggregation for Graph Nets
- Exphormer: Sparse Transformers for Graphs cites Recipe for a General, Powerful, Scalable Graph Transformer
- Exphormer: Sparse Transformers for Graphs cites Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Exphormer: Sparse Transformers for Graphs cites Structure-Aware Transformer for Graph Representation Learning
- Exphormer: Sparse Transformers for Graphs cites Rethinking Graph Transformers with Spectral Attention
- Exphormer: Sparse Transformers for Graphs cites Principal Neighbourhood Aggregation for Graph Nets
- Principal Neighbourhood Aggregation for Graph Nets cites Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Brain Network Transformer cites Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Brain Network Transformer cites Rethinking Graph Transformers with Spectral Attention
Cited by top-tier papers165
- Recipe for a General, Powerful, Scalable Graph TransformerLadislav Rampásek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu et al.NeurIPS 2022 · 1,216 citations
- Structure-Aware Transformer for Graph Representation LearningDexiong Chen, Leslie O'Bray, Karsten M. BorgwardtICML 2022 · 349 citations
- Brain Network TransformerXuan Kan, Wei Dai, Hejie Cui, Zilong Zhang et al.NeurIPS 2022 · 272 citations
- Federated Learning on Non-IID Graphs via Structural Knowledge SharingYue Tan, Yixin Liu, Guodong Long, Jing Jiang et al.AAAI 2023 · 224 citations
- Exphormer: Sparse Transformers for GraphsHamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland et al.ICML 2023 · 219 citations
Builds on13
- Open Graph Benchmark: Datasets for Machine Learning on GraphsWeihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong et al.NeurIPS 2020 · 3,935 citations
- DeepGCNs: Can GCNs Go As Deep As CNNs?Guohao Li, Matthias Müller, Ali K. Thabet, Bernard GhanemICCV 2019 · 1,586 citations
- Principal Neighbourhood Aggregation for Graph NetsGabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò et al.NeurIPS 2020 · 914 citations
- Rethinking Graph Transformers with Spectral AttentionDevin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau et al.NeurIPS 2021 · 854 citations
- How much Position Information Do Convolutional Neural Networks Encode?Md. Amirul Islam, Sen Jia, Neil D. B. BruceICLR 2020 · 392 citations
Related papers
- On the Stability of Expressive Positional Encodings for GraphsYinan Huang, William Lu, Joshua Robinson, Yu Yang et al.ICLR 2024 · 32 citations
- Graph Positional and Structural EncoderSemih Cantürk, Renming Liu, Olivier Lapointe-Gagné, Vincent Létourneau et al.ICML 2024 · 33 citations
- Equivariant and Stable Positional Encoding for More Powerful Graph Neural NetworksHaorui Wang, Haoteng Yin, Muhan Zhang, Pan LiICLR 2022 · 138 citations
- Learning Efficient Positional Encodings with Graph Neural NetworksCharilaos I. Kanatsoulis, Evelyn Choi, Stefanie Jegelka, Jure Leskovec et al.ICLR 2025
- LEAP: Local ECT-Based Learnable Positional Encodings for GraphsJuan Amboage, Ernst Röell, Patrick Schnider, Bastian RieckICLR 2026 · 3 citations
