https://arxiv.org/abs/2210.08122
Despite the enormous success of Graph Convolutional Networks (GCNs) in mod- elling graph-structured data, most of the current GCNs are shallow due to the notoriously challenging problems of over-smoothening and information squashing along with conventional difficulty caused by vanishing gradients and over-fitting. Previous works have been primarily focused on the study of over-smoothening and over-squashing phenomenon in training deep GCNs. Surprisingly, in comparison with CNNs/RNNs, very limited attention has been given towards understanding how healthy gradient flow can benefit the trainability of deep GCNs. In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the- art (SOTA) level performance from vanilla-GCNs [1]. Next, we argue that blindly adopting the Glorot initialization for GCNs is not optimal, and derive a topology- aware isometric initialization scheme for vanilla-GCNs based on the principles of isometry. Additionally, contrary to ad-hoc addition of skip-connections, we propose to use gradient-guided dynamic rewiring of vanilla-GCNs with skip- connections. Our dynamic rewiring method uses the gradient flow within each layer during training to introduce skip-connections on-demand basis. We provide extensive empirical evidence across multiple datasets that our methods improves gradient flow in deep vanilla-GCNs and significantly boost their performance to comfortably compete and outperform many fancy state-of-the-art methods.
If you find our work helpful in your research, please cite our paper
If you find our code implementation helpful for your own resarch or work, please cite our paper.
@inproceedings{Jaiswal22GradientGCN,
title={Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again},
author={Ajay Jaiswal, Peihao Wang, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang},
booktitle={NeurIPS 2022},
year={2022}
}