  We analyze the performance of feedforward vs. recurrent neural network (RNN)
architectures and associated training methods for learned frame prediction. To
this effect, we trained a residual fully convolutional neural network (FCNN), a
convolutional RNN (CRNN), and a convolutional long short-term memory (CLSTM)
network for next frame prediction using the mean square loss. We performed both
stateless and stateful training for recurrent networks. Experimental results
show that the residual FCNN architecture performs the best in terms of peak
signal to noise ratio (PSNR) at the expense of higher training and test
(inference) computational complexity. The CRNN can be trained stably and very
efficiently using the stateful truncated backpropagation through time
procedure, and it requires an order of magnitude less inference runtime to
achieve near real-time frame prediction with an acceptable performance.
