  Federated Learning (FL) is a framework which enables distributed model
training using a large corpus of decentralized training data. Existing methods
aggregate models disregarding their internal representations, which are crucial
for training models in vision tasks. System and statistical heterogeneity
(e.g., highly imbalanced and non-i.i.d. data) further harm model training. To
this end, we introduce a method, called FedProto, which computes client
deviations using margins of prototypical representations learned on distributed
data, and applies them to drive federated optimization via an attention
mechanism. In addition, we propose three methods to analyse statistical
properties of feature representations learned in FL, in order to elucidate the
relationship between accuracy, margins and feature discrepancy of FL models. In
experimental analyses, FedProto demonstrates state-of-the-art accuracy and
convergence rate across image classification and semantic segmentation
benchmarks by enabling maximum margin training of FL models. Moreover, FedProto
reduces uncertainty of predictions of FL models compared to the baseline. To
our knowledge, this is the first work evaluating FL models in dense prediction
tasks, such as semantic segmentation.
