  Unsupervised methods have proven effective for discriminative tasks in a
single-modality scenario. In this paper, we present a multimodal framework for
learning sparse representations that can capture semantic correlation between
modalities. The framework can model relationships at a higher level by forcing
the shared sparse representation. In particular, we propose the use of joint
dictionary learning technique for sparse coding and formulate the joint
representation for concision, cross-modal representations (in case of a missing
modality), and union of the cross-modal representations. Given the accelerated
growth of multimodal data posted on the Web such as YouTube, Wikipedia, and
Twitter, learning good multimodal features is becoming increasingly important.
We show that the shared representations enabled by our framework substantially
improve the classification performance under both unimodal and multimodal
settings. We further show how deep architectures built on the proposed
framework are effective for the case of highly nonlinear correlations between
modalities. The effectiveness of our approach is demonstrated experimentally in
image denoising, multimedia event detection and retrieval on the TRECVID
dataset (audio-video), category classification on the Wikipedia dataset
(image-text), and sentiment classification on PhotoTweet (image-text).
