  In this chapter, the regulation of Unmanned Aerial Vehicle (UAV)
communication network is investigated in the presence of dynamic changes in the
UAV lineup and user distribution. We target an optimal UAV control policy which
is capable of identifying the upcoming change in the UAV lineup (quit or
join-in) or user distribution, and proactively relocating the UAVs ahead of the
change rather than passively dispatching the UAVs after the change.
Specifically, a deep reinforcement learning (DRL)-based UAV control framework
is developed to maximize the accumulated user satisfaction (US) score for a
given time horizon which is able to handle the change in both the UAV lineup
and user distribution. The framework accommodates the changed dimension of the
state-action space before and after the UAV lineup change by deliberate state
transition design. In addition, to handle the continuous state and action
space, deep deterministic policy gradient (DDPG) algorithm, which is an
actor-critic based DRL, is exploited. Furthermore, to promote the learning
exploration around the timing of the change, the original DDPG is adapted into
an asynchronous parallel computing (APC) structure which leads to a better
training performance in both the critic and actor networks. Finally, extensive
simulations are conducted to validate the convergence of the proposed learning
approach, and demonstrate its capability in jointly handling the dynamics in
UAV lineup and user distribution as well as its superiority over a passive
reaction method.
