  Many methods for learning from video sequences involve temporally processing
2D CNN features from the individual frames or directly utilizing 3D
convolutions within high-performing 2D CNN architectures. The focus typically
remains on how to incorporate the temporal processing within an already stable
spatial architecture. This work constructs an MNIST-based video dataset with
parameters controlling relevant facets of common video-related tasks:
classification, ordering, and speed estimation. Models trained on this dataset
are shown to differ in key ways depending on the task and their use of 2D
convolutions, 3D convolutions, or convolutional LSTMs. An empirical analysis
indicates a complex, interdependent relationship between the spatial and
temporal dimensions with design choices having a large impact on a network's
ability to learn the appropriate spatiotemporal features.
