  This paper proposes a novel design on the wireless powered communication
network (WPCN) in dynamic environments under the assistance of multiple
unmanned aerial vehicles (UAVs). Unlike the existing studies, where the
low-power wireless nodes (WNs) often conform to the coherent
harvest-then-transmit protocol, under our newly proposed double-threshold based
WN type updating rule, each WN can dynamically and repeatedly update its WN
type as an E-node for non-linear energy harvesting over time slots or an I-node
for transmitting data over sub-slots. To maximize the total transmission data
size of all the WNs over T slots, each of the UAVs individually determines its
trajectory and binary wireless energy transmission (WET) decisions over times
slots and its binary wireless data collection (WDC) decisions over sub-slots,
under the constraints of each UAV's limited on-board energy and each WN's node
type updating rule. However, due to the UAVs' tightly-coupled trajectories with
their WET and WDC decisions, as well as each WN's time-varying battery energy,
this problem is difficult to solve optimally. We then propose a new multi-agent
based hierarchical deep reinforcement learning (MAHDRL) framework with two
tiers to solve the problem efficiently, where the soft actor critic (SAC)
policy is designed in tier-1 to determine each UAV's continuous trajectory and
binary WET decision over time slots, and the deep-Q learning (DQN) policy is
designed in tier-2 to determine each UAV's binary WDC decisions over sub-slots
under the given UAV trajectory from tier-1. Both of the SAC policy and the DQN
policy are executed distributively at each UAV. Finally, extensive simulation
results are provided to validate the outweighed performance of the proposed
MAHDRL approach over various state-of-the-art benchmarks.
