  Federated Learning (FL) has been widely accepted as the solution for
privacy-preserving machine learning without collecting raw data. While new
technologies proposed in the past few years do evolve the FL area,
unfortunately, the evaluation results presented in these works fall short in
integrity and are hardly comparable because of the inconsistent evaluation
metrics and experimental settings. In this paper, we propose a holistic
evaluation framework for FL called FedEval, and present a benchmarking study on
seven state-of-the-art FL algorithms. Specifically, we first introduce the core
evaluation taxonomy model, called FedEval-Core, which covers four essential
evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency,
with various well-defined metrics and experimental settings. Based on the
FedEval-Core, we further develop an FL evaluation platform with standardized
evaluation settings and easy-to-use interfaces. We then provide an in-depth
benchmarking study between the seven well-known FL algorithms, including
FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensively
analyze the advantages and disadvantages of these algorithms and further
identify the suitable practical scenarios for different algorithms, which is
rarely done by prior work. Lastly, we excavate a set of take-away insights and
future research directions, which are very helpful for researchers in the FL
area.
