  In this paper, we employ multiple wireless-powered relays to assist
information transmission from a multi-antenna access point to a single-antenna
receiver. The wireless relays can operate in either the passive mode via
backscatter communications or the active mode via RF communications, depending
on their channel conditions and energy states. We aim to maximize the overall
throughput by jointly optimizing the access point's beamforming and the relays'
radio modes and operating parameters. Due to the non-convex and combinatorial
structure, we develop a novel optimization-driven hierarchical deep
deterministic policy gradient (H-DDPG) approach to adapt the beamforming and
relay strategies dynamically. The optimization-driven H-DDPG algorithm firstly
decomposes the binary relay mode selection into the outer-loop deep Q-network
(DQN) algorithm and then optimizes the continuous beamforming and relaying
parameters by using the inner-loop DDPG algorithm. Secondly, to improve the
learning efficiency, we integrate the model-based optimization into the DDPG
framework by providing a better-informed target estimation for DNN training.
Simulation results reveal that these two special designs ensure a more stable
learning and achieve a higher reward performance, up to nearly 20%, compared to
the conventional DDPG approach.
