  Federated learning (FL) facilitates distributed training across different IoT
and edge devices, safeguarding the privacy of their data. The inherent
distributed structure of FL introduces vulnerabilities, especially from
adversarial devices aiming to skew local updates to their advantage. Despite
the plethora of research focusing on Byzantine-resilient FL, the academic
community has yet to establish a comprehensive benchmark suite, pivotal for
impartial assessment and comparison of different techniques. This paper
presents Blades, a scalable, extensible, and easily configurable benchmark
suite that supports researchers and developers in efficiently implementing and
validating novel strategies against baseline algorithms in Byzantine-resilient
FL. Blades contains built-in implementations of representative attack and
defense strategies and offers a user-friendly interface that seamlessly
integrates new ideas. Using Blades, we re-evaluate representative attacks and
defenses on wide-ranging experimental configurations (approximately 1,500
trials in total). Through our extensive experiments, we gained new insights
into FL robustness and highlighted previously overlooked limitations due to the
absence of thorough evaluations and comparisons of baselines under various
attack settings.
