  Predicting the outcomes of integrating Unmanned Aerial Systems (UAS) into the
National Airspace System (NAS) is a complex problem which is required to be
addressed by simulation studies before allowing the routine access of UAS into
the NAS. This paper focuses on providing a 3-dimensional (3D) simulation
framework using a game theoretical methodology to evaluate integration concepts
using scenarios where manned and unmanned air vehicles co-exist. In the
proposed method, human pilot interactive decision making process is
incorporated into airspace models which can fill the gap in the literature
where the pilot behavior is generally assumed to be known a priori. The
proposed human pilot behavior is modeled using dynamic level-k reasoning
concept and approximate reinforcement learning. The level-k reasoning concept
is a notion in game theory and is based on the assumption that humans have
various levels of decision making. In the conventional "static" approach, each
agent makes assumptions about his or her opponents and chooses his or her
actions accordingly. On the other hand, in the dynamic level-k reasoning,
agents can update their beliefs about their opponents and revise their level-k
rule. In this study, Neural Fitted Q Iteration, which is an approximate
reinforcement learning method, is used to model time-extended decisions of
pilots with 3D maneuvers. An analysis of UAS integration is conducted using an
example 3D scenario in the presence of manned aircraft and fully autonomous UAS
equipped with sense and avoid algorithms.
