  This paper presents an unsupervised approach that leverages raw aerial videos
to learn to estimate planar homographic transformation between consecutive
video frames. Previous learning-based estimators work on pairs of images to
estimate their planar homographic transformations but suffer from severe
over-fitting issues, especially when applying over aerial videos. To address
this concern, we develop a sequential estimator that directly processes a
sequence of video frames and estimates their pairwise planar homographic
transformations in batches. We also incorporate a set of spatial-temporal
knowledge to regularize the learning of such a sequence-to-sequence model. We
collect a set of challenging aerial videos and compare the proposed method to
the alternative algorithms. Empirical studies suggest that our sequential model
achieves significant improvement over alternative image-based methods and the
knowledge-rich regularization further boosts our system performance. Our codes
and dataset could be found at https://github.com/Paul-LiPu/DeepVideoHomography
