  Data generated from real world events are usually temporal and contain
multimodal information such as audio, visual, depth, sensor etc. which are
required to be intelligently combined for classification tasks. In this paper,
we propose a novel generalized deep neural network architecture where temporal
streams from multiple modalities are combined. There are total M+1 (M is the
number of modalities) components in the proposed network. The first component
is a novel temporally hybrid Recurrent Neural Network (RNN) that exploits the
complimentary nature of the multimodal temporal information by allowing the
network to learn both modality specific temporal dynamics as well as the
dynamics in a multimodal feature space. M additional components are added to
the network which extract discriminative but non-temporal cues from each
modality. Finally, the predictions from all of these components are linearly
combined using a set of automatically learned weights. We perform exhaustive
experiments on three different datasets spanning four modalities. The proposed
network is relatively 3.5%, 5.7% and 2% better than the best performing
temporal multimodal baseline for UCF-101, CCV and Multimodal Gesture datasets
respectively.
