  In this paper, we propose a two-layer framework to learn the optimal handover
(HO) controllers in possibly large-scale wireless systems supporting mobile
Internet-of-Things (IoT) users or traditional cellular users, where the user
mobility patterns could be heterogeneous. In particular, our proposed framework
first partitions the user equipments (UEs) with different mobility patterns
into clusters, where the mobility patterns are similar in the same cluster.
Then, within each cluster, an asynchronous multi-user deep reinforcement
learning scheme is developed to control the HO processes across the UEs in each
cluster, in the goal of lowering the HO rate while ensuring certain system
throughput. In this scheme, we use a deep neural network (DNN) as an HO
controller learned by each UE via reinforcement learning in a collaborative
fashion. Moreover, we use supervised learning in initializing the DNN
controller before the execution of reinforcement learning to exploit what we
already know with traditional HO schemes and to mitigate the negative effects
of random exploration at the initial stage. Furthermore, we show that the
adopted global-parameter-based asynchronous framework enables us to train
faster with more UEs, which could nicely address the scalability issue to
support large systems. Finally, simulation results demonstrate that the
proposed framework can achieve better performance than the state-of-art on-line
schemes, in terms of HO rates.
