  A Graph Convolutional Network (GCN) stacks several layers and in each layer
performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN)
for learning node representations over graph-structured data. Though powerful,
GCNs tend to suffer performance drop when the model gets deep. Previous works
focus on PROPs to study and mitigate this issue, but the role of TRANs is
barely investigated. In this work, we study performance degradation of GCNs by
experimentally examining how stacking only TRANs or PROPs works. We find that
TRANs contribute significantly, or even more than PROPs, to declining
performance, and moreover that they tend to amplify node-wise feature variance
in GCNs, causing variance inflammation that we identify as a key factor for
causing performance drop. Motivated by such observations, we propose a
variance-controlling technique termed Node Normalization (NodeNorm), which
scales each node's features using its own standard deviation. Experimental
results validate the effectiveness of NodeNorm on addressing performance
degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow
ones in cases where deep models are needed, and to achieve comparable results
with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and
can well generalize to other GNN architectures. Code is publicly available at
https://github.com/miafei/NodeNorm.
