  We propose a new way of incorporating temporal information present in videos
into Spatial Convolutional Neural Networks (ConvNets) trained on images, that
avoids training Spatio-Temporal ConvNets from scratch. We describe several
initializations of weights in 3D Convolutional Layers of Spatio-Temporal
ConvNet using 2D Convolutional Weights learned from ImageNet. We show that it
is important to initialize 3D Convolutional Weights judiciously in order to
learn temporal representations of videos. We evaluate our methods on the
UCF-101 dataset and demonstrate improvement over Spatial ConvNets.
