  Sheet music, audio, and lyrics are three main modalities during writing a
song. In this paper, we propose an unsupervised generative adversarial
alignment representation (UGAAR) model to learn deep discriminative
representations shared across three major musical modalities: sheet music,
lyrics, and audio, where a deep neural network based architecture on three
branches is jointly trained. In particular, the proposed model can transfer the
strong relationship between audio and sheet music to audio-lyrics and
sheet-lyrics pairs by learning the correlation in the latent shared subspace.
We apply CCA components of audio and sheet music to establish new ground truth.
The generative (G) model learns the correlation of two couples of transferred
pairs to generate new audio-sheet pair for a fixed lyrics to challenge the
discriminative (D) model. The discriminative model aims at distinguishing the
input which is from the generative model or the ground truth. The two models
simultaneously train in an adversarial way to enhance the ability of deep
alignment representation learning. Our experimental results demonstrate the
feasibility of our proposed UGAAR for alignment representation learning among
sheet music, audio, and lyrics.
