  Graph Convolutional Networks (GCNs) have emerged as powerful tools for
learning on network structured data. Although empirically successful, GCNs
exhibit certain behaviour that has no rigorous explanation -- for instance, the
performance of GCNs significantly degrades with increasing network depth,
whereas it improves marginally with depth using skip connections. This paper
focuses on semi-supervised learning on graphs, and explains the above
observations through the lens of Neural Tangent Kernels (NTKs). We derive NTKs
corresponding to infinitely wide GCNs (with and without skip connections).
Subsequently, we use the derived NTKs to identify that, with suitable
normalisation, network depth does not always drastically reduce the performance
of GCNs -- a fact that we also validate through extensive simulation.
Furthermore, we propose NTK as an efficient `surrogate model' for GCNs that
does not suffer from performance fluctuations due to hyper-parameter tuning
since it is a hyper-parameter free deterministic kernel. The efficacy of this
idea is demonstrated through a comparison of different skip connections for
GCNs using the surrogate NTKs.
