  In conventional distributed learning over a network, multiple agents
collaboratively build a common machine learning model. However, due to the
underlying non-i.i.d. data distribution among agents, the unified learning
model becomes inefficient for each agent to process its locally accessible
data. To address this problem, we propose a graph-attention-based personalized
training algorithm (GATTA) for distributed deep learning. The GATTA enables
each agent to train its local personalized model while exploiting its
correlation with neighboring nodes and utilizing their useful information for
aggregation. In particular, the personalized model in each agent is composed of
a global part and a node-specific part. By treating each agent as one node in a
graph and the node-specific parameters as its features, the benefits of the
graph attention mechanism can be inherited. Namely, instead of aggregation
based on averaging, it learns the specific weights for different neighboring
nodes without requiring prior knowledge about the graph structure or the
neighboring nodes' data distribution. Furthermore, relying on the
weight-learning procedure, we develop a communication-efficient GATTA by
skipping the transmission of information with small aggregation weights.
Additionally, we theoretically analyze the convergence properties of GATTA for
non-convex loss functions. Numerical results validate the excellent
performances of the proposed algorithms in terms of convergence and
communication cost.
