  Recent years have witnessed great success in handling graph-related tasks
with Graph Neural Networks (GNNs). Despite their great academic success,
Multi-Layer Perceptrons (MLPs) remain the primary workhorse for practical
industrial applications. One reason for this academic-industrial gap is the
neighborhood-fetching latency incurred by data dependency in GNNs, which make
it hard to deploy for latency-sensitive applications that require fast
inference. Conversely, without involving any feature aggregation, MLPs have no
data dependency and infer much faster than GNNs, but their performance is less
competitive. Motivated by these complementary strengths and weaknesses, we
propose a Graph Self-Distillation on Neighborhood (GSDN) framework to reduce
the gap between GNNs and MLPs. Specifically, the GSDN framework is based purely
on MLPs, where structural information is only implicitly used as prior to guide
knowledge self-distillation between the neighborhood and the target,
substituting the explicit neighborhood information propagation as in GNNs. As a
result, GSDN enjoys the benefits of graph topology-awareness in training but
has no data dependency in inference. Extensive experiments have shown that the
performance of vanilla MLPs can be greatly improved with self-distillation,
e.g., GSDN improves over stand-alone MLPs by 15.54% on average and outperforms
the state-of-the-art GNNs on six datasets. Regarding inference speed, GSDN
infers 75X-89X faster than existing GNNs and 16X-25X faster than other
inference acceleration methods.
