  GNN-to-MLP distillation aims to utilize knowledge distillation (KD) to learn
computationally-efficient multi-layer perceptron (student MLP) on graph data by
mimicking the output representations of teacher GNN. Existing methods mainly
make the MLP to mimic the GNN predictions over a few class labels. However, the
class space may not be expressive enough for covering numerous diverse local
graph structures, thus limiting the performance of knowledge transfer from GNN
to MLP. To address this issue, we propose to learn a new powerful graph
representation space by directly labeling nodes' diverse local structures for
GNN-to-MLP distillation. Specifically, we propose a variant of VQ-VAE to learn
a structure-aware tokenizer on graph data that can encode each node's local
substructure as a discrete code. The discrete codes constitute a codebook as a
new graph representation space that is able to identify different local graph
structures of nodes with the corresponding code indices. Then, based on the
learned codebook, we propose a new distillation target, namely soft code
assignments, to directly transfer the structural knowledge of each node from
GNN to MLP. The resulting framework VQGraph achieves new state-of-the-art
performance on GNN-to-MLP distillation in both transductive and inductive
settings across seven graph datasets. We show that VQGraph with better
performance infers faster than GNNs by 828x, and also achieves accuracy
improvement over GNNs and stand-alone MLPs by 3.90% and 28.05% on average,
respectively. Code: https://github.com/YangLing0818/VQGraph.
