  Ground-air negotiation via speech communication is a vital prerequisite for
ensuring safety and efficiency in air traffic control (ATC) operations.
However, with the increase in traffic flow, incorrect instructions caused by
human factors bring a great threat to ATC safety. Existing flight trajectory
prediction (FTP) approaches primarily rely on the flight status of historical
trajectory, leading to significant delays in the prediction of real-time
maneuvering instruction, which is not conducive to conflict detection. A major
reason is that spoken instructions and flight trajectories are presented in
different modalities in the current air traffic control (ATC) system, bringing
great challenges to considering the maneuvering instruction in the FTP tasks.
In this paper, a spoken instruction-aware FTP framework, called SIA-FTP, is
innovatively proposed to support high-maneuvering FTP tasks by incorporating
instant spoken instruction. To address the modality gap and minimize the data
requirements, a 3-stage learning paradigm is proposed to implement the SIA-FTP
framework in a progressive manner, including trajectory-based FTP pretraining,
intent-oriented instruction embedding learning, and multi-modal finetuning.
Specifically, the FTP model and the instruction embedding with maneuvering
semantics are pre-trained using volumes of well-resourced trajectory and text
data in the 1st and 2nd stages. In succession, a multi-modal fusion strategy is
proposed to incorporate the pre-trained instruction embedding into the FTP
model and integrate the two pre-trained networks into a joint model. Finally,
the joint model is finetuned using the limited trajectory-instruction data to
enhance the FTP performance within maneuvering instruction scenarios. The
experimental results demonstrated that the proposed framework presents an
impressive performance improvement in high-maneuvering scenarios.
