  A novel framework is proposed for cellular offloading with the aid of
multiple unmanned aerial vehicles (UAVs), while non-orthogonal multiple access
(NOMA) technique is employed at each UAV to further improve the spectrum
efficiency of the wireless network. The optimization problem of joint
three-dimensional (3D) trajectory design and power allocation is formulated for
maximizing the throughput. In an effort to solve this pertinent dynamic
problem, a K-means based clustering algorithm is first adopted for periodically
partitioning users. Afterward, a mutual deep Q-network (MDQN) algorithm is
proposed to jointly determine the optimal 3D trajectory and power allocation of
UAVs. In contrast to the conventional deep Q-network (DQN) algorithm, the MDQN
algorithm enables the experience of multi-agent to be input into a shared
neural network to shorten the training time with the assistance of state
abstraction. Numerical results demonstrate that: 1) the proposed MDQN algorithm
has a faster convergence rate than the conventional DQN algorithm in the
multi-agent case; 2) The achievable sum rate of the NOMA enhanced UAV network
is $23\%$ superior to the case of orthogonal multiple access (OMA); 3) By
designing the optimal 3D trajectory of UAVs with the aid of the MDON algorithm,
the sum rate of the network enjoys ${142\%}$ and ${56\%}$ gains than that of
invoking the circular trajectory and the 2D trajectory, respectively.
