  Privacy and security concerns in real-world applications have led to the
development of adversarially robust federated models. However, the
straightforward combination between adversarial training and federated learning
in one framework can lead to the undesired robustness deterioration. We
discover that the attribution behind this phenomenon is that the generated
adversarial data could exacerbate the data heterogeneity among local clients,
making the wrapped federated learning perform poorly. To deal with this
problem, we propose a novel framework called Slack Federated Adversarial
Training (SFAT), assigning the client-wise slack during aggregation to combat
the intensified heterogeneity. Theoretically, we analyze the convergence of the
proposed method to properly relax the objective when combining federated
learning and adversarial training. Experimentally, we verify the rationality
and effectiveness of SFAT on various benchmarked and real-world datasets with
different adversarial training and federated optimization methods. The code is
publicly available at https://github.com/ZFancy/SFAT.
