  Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD)
variant, has gained widespread popularity in federated learning (FL) due to its
fast convergence. However, federated Adam (FedAdam) algorithms suffer from a
threefold increase in uplink communication overhead compared to federated SGD
(FedSGD) algorithms, which arises from the necessity to transmit both local
model updates and first and second moment estimates from distributed devices to
the centralized server for aggregation. Driven by this issue, we propose a
novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices
sparsify the updates of local model parameters and moment estimates and
subsequently upload the sparse representations to the centralized server. To
further reduce the communication overhead, the updates of local model
parameters and moment estimates incorporate a shared sparse mask (SSM) into the
sparsification process, eliminating the need for three separate sparse masks.
Theoretically, we develop an upper bound on the divergence between the local
model trained by FedAdam-SSM and the desired model trained by centralized Adam,
which is related to sparsification error and imbalanced data distribution. By
minimizing the divergence bound between the model trained by FedAdam-SSM and
centralized Adam, we optimize the SSM to mitigate the learning performance
degradation caused by sparsification error. Additionally, we provide
convergence bounds for FedAdam-SSM in both convex and non-convex objective
function settings, and investigate the impact of local epoch, learning rate and
sparsification ratio on the convergence rate of FedAdam-SSM. Experimental
results show that FedAdam-SSM outperforms baselines in terms of convergence
rate (over 1.1$\times$ faster than the sparse FedAdam baselines) and test
accuracy (over 14.5\% ahead of the quantized FedAdam baselines).
