  This paper describes a network that captures multimodal correlations over
arbitrary timestamps. The proposed scheme operates as a complementary, extended
network over a multimodal convolutional neural network (CNN). Spatial and
temporal streams are required for action recognition by a deep CNN, but
overfitting reduction and fusing these two streams remain open problems. The
existing fusion approach averages the two streams. Here we propose a
correlation network with a Shannon fusion for learning a pre-trained CNN. A
Long-range video may consist of spatiotemporal correlations over arbitrary
times, which can be captured by forming the correlation network from simple
fully connected layers. This approach was found to complement the existing
network fusion methods. The importance of multimodal correlation is validated
in comparison experiments on the UCF-101 and HMDB-51 datasets. The multimodal
correlation enhanced the accuracy of the video recognition results.
