  In unmanned aerial vehicle (UAV) applications, the UAV's limited energy
supply and storage have triggered the development of intelligent
energy-conserving scheduling solutions. In this paper, we investigate energy
minimization for UAV-aided communication networks by jointly optimizing
data-transmission scheduling and UAV hovering time. The formulated problem is
combinatorial and non-convex with bilinear constraints. To tackle the problem,
firstly, we provide an optimal relax-and-approximate solution and develop a
near-optimal algorithm. Both the proposed solutions are served as offline
performance benchmarks but might not be suitable for online operation. To this
end, we develop a solution from a deep reinforcement learning (DRL) aspect. The
conventional RL/DRL, e.g., deep Q-learning, however, is limited in dealing with
two main issues in constrained combinatorial optimization, i.e., exponentially
increasing action space and infeasible actions. The novelty of solution
development lies in handling these two issues. To address the former, we
propose an actor-critic-based deep stochastic online scheduling (AC-DSOS)
algorithm and develop a set of approaches to confine the action space. For the
latter, we design a tailored reward function to guarantee the solution
feasibility. Numerical results show that, by consuming equal magnitude of time,
AC-DSOS is able to provide feasible solutions and saves 29.94% energy compared
with a conventional deep actor-critic method. Compared to the developed
near-optimal algorithm, AC-DSOS consumes around 10% higher energy but reduces
the computational time from minute-level to millisecond-level.
