  Graph neural networks (GNNs) have received tremendous attention due to their
superiority in learning node representations. These models rely on message
passing and feature transformation functions to encode the structural and
feature information from neighbors. However, stacking more convolutional layers
significantly decreases the performance of GNNs. Most recent studies attribute
this limitation to the over-smoothing issue, where node embeddings converge to
indistinguishable vectors. Through a number of experimental observations, we
argue that the main factor degrading the performance is the unstable forward
normalization and backward gradient resulted from the improper design of the
feature transformation, especially for shallow GNNs where the over-smoothing
has not happened. Therefore, we propose a novel orthogonal feature
transformation, named Ortho-GConv, which could generally augment the existing
GNN backbones to stabilize the model training and improve the model's
generalization performance. Specifically, we maintain the orthogonality of the
feature transformation comprehensively from three perspectives, namely hybrid
weight initialization, orthogonal transformation, and orthogonal
regularization. By equipping the existing GNNs (e.g. GCN, JKNet, GCNII) with
Ortho-GConv, we demonstrate the generality of the orthogonal feature
transformation to enable stable training, and show its effectiveness for node
and graph classification tasks.
