  Unmanned combat air vehicle (UCAV) combat is a challenging scenario with
continuous action space. In this paper, we propose a general hierarchical
framework to resolve the within-vision-range (WVR) air-to-air combat problem
under 6 dimensions of degree (6-DOF) dynamics. The core idea is to divide the
whole decision process into two loops and use reinforcement learning (RL) to
solve them separately. The outer loop takes into account the current combat
situation and decides the expected macro behavior of the aircraft according to
a combat strategy. Then the inner loop tracks the macro behavior with a flight
controller by calculating the actual input signals for the aircraft. We design
the Markov decision process for both the outer loop strategy and inner loop
controller, and train them by proximal policy optimization (PPO) algorithm. For
the inner loop controller, we design an effective reward function to accurately
track various macro behavior. For the outer loop strategy, we further adopt a
fictitious self-play mechanism to improve the combat performance by constantly
combating against the historical strategies. Experiment results show that the
inner loop controller can achieve better tracking performance than fine-tuned
PID controller, and the outer loop strategy can perform complex maneuvers to
get higher and higher winning rate, with the generation evolves.
