  While recurrent neural networks (RNNs) demonstrate outstanding capabilities
for future video frame prediction, they model dynamics in a discrete time
space, i.e., they predict the frames sequentially with a fixed temporal step.
RNNs are therefore prone to accumulate the error as the number of future frames
increases. In contrast, partial differential equations (PDEs) model physical
phenomena like dynamics in a continuous time space. However, the estimated PDE
for frame forecasting needs to be numerically solved, which is done by
discretization of the PDE and diminishes most of the advantages compared to
discrete models. In this work, we, therefore, propose to approximate the motion
in a video by a continuous function using the Taylor series. To this end, we
introduce TaylorSwiftNet, a novel convolutional neural network that learns to
estimate the higher order terms of the Taylor series for a given input video.
TaylorSwiftNet can swiftly predict future frames in parallel and it allows to
change the temporal resolution of the forecast frames on-the-fly. The
experimental results on various datasets demonstrate the superiority of our
model.
