  Federated Learning (FL) is a promising research paradigm that enables the
collaborative training of machine learning models among various parties without
the need for sensitive information exchange. Nonetheless, retaining data in
individual clients introduces fundamental challenges to achieving performance
on par with centrally trained models. Our study provides an extensive review of
federated learning applied to visual recognition. It underscores the critical
role of thoughtful architectural design choices in achieving optimal
performance, a factor often neglected in the FL literature. Many existing FL
solutions are tested on shallow or simple networks, which may not accurately
reflect real-world applications. This practice restricts the transferability of
research findings to large-scale visual recognition models. Through an in-depth
analysis of diverse cutting-edge architectures such as convolutional neural
networks, transformers, and MLP-mixers, we experimentally demonstrate that
architectural choices can substantially enhance FL systems' performance,
particularly when handling heterogeneous data. We study 19 visual recognition
models from five different architectural families on four challenging FL
datasets. We also re-investigate the inferior performance of convolution-based
architectures in the FL setting and analyze the influence of normalization
layers on the FL performance. Our findings emphasize the importance of
architectural design for computer vision tasks in practical scenarios,
effectively narrowing the performance gap between federated and centralized
learning. Our source code is available at
https://github.com/sarapieri/fed_het.git.
