  Unmanned aerial vehicles (UAVs) are the emerging vital components of
millimeter wave (mmWave) wireless systems. Accurate beam alignment is essential
for efficient beam-based mmWave communications of UAVs with base stations
(BSs). Conventional beam sweeping approaches often have large overhead due to
the high mobility and autonomous operation of UAVs. Learning-based approaches
greatly reduce the overhead by leveraging UAV data, like position to identify
optimal beam directions. In this paper, we propose a reinforcement learning
(RL)-based framework for UAV-BS beam alignment using deep Q-Network (DQN) in a
mmWave setting. We consider uplink communications where the UAV hovers around
5G new radio (NR) BS coverage area, with varying channel conditions. The
proposed learning framework uses the location information to maximize data rate
through the optimal beam-pairs efficiently, upon every communication request
from UAV inside the multi-location environment. We compare our proposed
framework against the Multi-Armed Bandit (MAB) learning-based approach and the
traditional exhaustive approach, respectively, and also analyse the training
performance of DQN-based beam alignment over different coverage area
requirements and channel conditions. Our results show that the proposed
DQN-based beam alignment converges faster and generic for different
environmental conditions. The framework can also learn optimal beam alignment
comparable to the exhaustive approach in an online manner under real-time
conditions.
