  Graph Attention Networks (GATs) are the state-of-the-art neural architecture
for representation learning with graphs. GATs learn attention functions that
assign weights to nodes so that different nodes have different influences in
the feature aggregation steps. In practice, however, induced attention
functions are prone to over-fitting due to the increasing number of parameters
and the lack of direct supervision on attention weights. GATs also suffer from
over-smoothing at the decision boundary of nodes. Here we propose a framework
to address their weaknesses via margin-based constraints on attention during
training. We first theoretically demonstrate the over-smoothing behavior of
GATs and then develop an approach using constraint on the attention weights
according to the class boundary and feature aggregation pattern. Furthermore,
to alleviate the over-fitting problem, we propose additional constraints on the
graph structure. Extensive experiments and ablation studies on common benchmark
datasets demonstrate the effectiveness of our method, which leads to
significant improvements over the previous state-of-the-art graph attention
methods on all datasets.
