  End-to-end (E2E) spoken language understanding (SLU) systems can infer the
semantics of a spoken utterance directly from an audio signal. However,
training an E2E system remains a challenge, largely due to the scarcity of
paired audio-semantics data. In this paper, we treat an E2E system as a
multi-modal model, with audio and text functioning as its two modalities, and
use a cross-modal latent space (CMLS) architecture, where a shared latent space
is learned between the `acoustic' and `text' embeddings. We propose using
different multi-modal losses to explicitly guide the acoustic embeddings to be
closer to the text embeddings, obtained from a semantically powerful
pre-trained BERT model. We train the CMLS model on two publicly available E2E
datasets, across different cross-modal losses and show that our proposed
triplet loss function achieves the best performance. It achieves a relative
improvement of 1.4% and 4% respectively over an E2E model without a cross-modal
space and a relative improvement of 0.7% and 1% over a previously published
CMLS model using $L_2$ loss. The gains are higher for a smaller, more
complicated E2E dataset, demonstrating the efficacy of using an efficient
cross-modal loss function, especially when there is limited E2E training data
available.
