  This paper presents and characterizes an Open Application Repository for
Federated Learning (OARF), a benchmark suite for federated machine learning
systems. Previously available benchmarks for federated learning have focused
mainly on synthetic datasets and use a limited number of applications. OARF
mimics more realistic application scenarios with publicly available data sets
as different data silos in image, text and structured data. Our
characterization shows that the benchmark suite is diverse in data size,
distribution, feature distribution and learning task complexity. The extensive
evaluations with reference implementations show the future research
opportunities for important aspects of federated learning systems. We have
developed reference implementations, and evaluated the important aspects of
federated learning, including model accuracy, communication cost, throughput
and convergence time. Through these evaluations, we discovered some interesting
findings such as federated learning can effectively increase end-to-end
throughput.
