  Unmanned aerial vehicles (UAVs) are now widely applied to data acquisition
due to its low cost and fast mobility. With the increasing volume of aerial
videos, the demand for automatically parsing these videos is surging. To
achieve this, current researches mainly focus on extracting a holistic feature
with convolutions along both spatial and temporal dimensions. However, these
methods are limited by small temporal receptive fields and cannot adequately
capture long-term temporal dependencies which are important for describing
complicated dynamics. In this paper, we propose a novel deep neural network,
termed FuTH-Net, to model not only holistic features, but also temporal
relations for aerial video classification. Furthermore, the holistic features
are refined by the multi-scale temporal relations in a novel fusion module for
yielding more discriminative video representations. More specially, FuTH-Net
employs a two-pathway architecture: (1) a holistic representation pathway to
learn a general feature of both frame appearances and shortterm temporal
variations and (2) a temporal relation pathway to capture multi-scale temporal
relations across arbitrary frames, providing long-term temporal dependencies.
Afterwards, a novel fusion module is proposed to spatiotemporal integrate the
two features learned from the two pathways. Our model is evaluated on two
aerial video classification datasets, ERA and Drone-Action, and achieves the
state-of-the-art results. This demonstrates its effectiveness and good
generalization capacity across different recognition tasks (event
classification and human action recognition). To facilitate further research,
we release the code at https://gitlab.lrz.de/ai4eo/reasoning/futh-net.
