  Graph neural networks (GNN) suffer from severe inefficiency. It is mainly
caused by the exponential growth of node dependency with the increase of
layers. It extremely limits the application of stochastic optimization
algorithms so that the training of GNN is usually time-consuming. To address
this problem, we propose to decouple a multi-layer GNN as multiple simple
modules for more efficient training, which is comprised of classical forward
training (FT)and designed backward training (BT). Under the proposed framework,
each module can be trained efficiently in FT by stochastic algorithms without
distortion of graph information owing to its simplicity. To avoid the only
unidirectional information delivery of FT and sufficiently train shallow
modules with the deeper ones, we develop a backward training mechanism that
makes the former modules perceive the latter modules. The backward training
introduces the reversed information delivery into the decoupled modules as well
as the forward information delivery. To investigate how the decoupling and
greedy training affect the representational capacity, we theoretically prove
that the error produced by linear modules will not accumulate on unsupervised
tasks in most cases. The theoretical and experimental results show that the
proposed framework is highly efficient with reasonable performance.
