  Latest federated learning (FL) methods started to focus on how to use
unlabeled data in clients for training due to users' privacy concerns, high
labeling costs, or lack of expertise. However, current Federated
Semi-Supervised/Self-Supervised Learning (FSSL) approaches fail to learn
large-scale images because of the limited computing resources of local clients.
In this paper, we introduce a new framework FedMAE, which stands for Federated
Masked AutoEncoder, to address the problem of how to utilize unlabeled
large-scale images for FL. Specifically, FedMAE can pre-train one-block Masked
AutoEncoder (MAE) using large images in lightweight client devices, and then
cascades multiple pre-trained one-block MAEs in the server to build a
multi-block ViT backbone for downstream tasks. Theoretical analysis and
experimental results on image reconstruction and classification show that our
FedMAE achieves superior performance compared to the state-of-the-art FSSL
methods.
