  Little research focuses on cross-modal correlation learning where temporal
structures of different data modalities such as audio and lyrics are taken into
account. Stemming from the characteristic of temporal structures of music in
nature, we are motivated to learn the deep sequential correlation between audio
and lyrics. In this work, we propose a deep cross-modal correlation learning
architecture involving two-branch deep neural networks for audio modality and
text modality (lyrics). Different modality data are converted to the same
canonical space where inter modal canonical correlation analysis is utilized as
an objective function to calculate the similarity of temporal structures. This
is the first study on understanding the correlation between language and music
audio through deep architectures for learning the paired temporal correlation
of audio and lyrics. Pre-trained Doc2vec model followed by fully-connected
layers (fully-connected deep neural network) is used to represent lyrics. Two
significant contributions are made in the audio branch, as follows: i)
pre-trained CNN followed by fully-connected layers is investigated for
representing music audio. ii) We further suggest an end-to-end architecture
that simultaneously trains convolutional layers and fully-connected layers to
better learn temporal structures of music audio. Particularly, our end-to-end
deep architecture contains two properties: simultaneously implementing feature
learning and cross-modal correlation learning, and learning joint
representation by considering temporal structures. Experimental results, using
audio to retrieve lyrics or using lyrics to retrieve audio, verify the
effectiveness of the proposed deep correlation learning architectures in
cross-modal music retrieval.
