  Federated learning (FL) enables collaborative model training among
distributed devices without data sharing, but existing FL suffers from poor
scalability because of global model synchronization. To address this issue,
hierarchical federated learning (HFL) has been recently proposed to let edge
servers aggregate models of devices in proximity, while synchronizing via the
cloud periodically. However, a critical open challenge about how to make a good
synchronization scheme (when devices and edges should be synchronized) is still
unsolved. Devices are heterogeneous in computing and communication capability,
and their data could be non-IID. No existing work can well synchronize various
roles (\textit{e.g.}, devices and edges) in HFL to guarantee high learning
efficiency and accuracy. In this paper, we propose a learning-based
synchronization scheme for HFL systems. By collecting data such as edge models,
CPU usage, communication time, \textit{etc}., we design a deep reinforcement
learning-based approach to decide the frequencies of cloud aggregation and edge
aggregation, respectively. The proposed scheme well considers device
heterogeneity, non-IID data and device mobility, to maximize the training model
accuracy while minimizing the energy overhead. Meanwhile, the convergence bound
of the proposed synchronization scheme has been analyzed. And we build an HFL
testbed and conduct the experiments with real data obtained from Raspberry Pi
and Alibaba Cloud. Extensive experiments under various settings are conducted
to confirm the effectiveness of \textit{Arena}.
