  Recent research has shown that word embedding spaces learned from text
corpora of different languages can be aligned without any parallel data
supervision. Inspired by the success in unsupervised cross-lingual word
embeddings, in this paper we target learning a cross-modal alignment between
the embedding spaces of speech and text learned from corpora of their
respective modalities in an unsupervised fashion. The proposed framework learns
the individual speech and text embedding spaces, and attempts to align the two
spaces via adversarial training, followed by a refinement procedure. We show
how our framework could be used to perform spoken word classification and
translation, and the results on these two tasks demonstrate that the
performance of our unsupervised alignment approach is comparable to its
supervised counterpart. Our framework is especially useful for developing
automatic speech recognition (ASR) and speech-to-text translation systems for
low- or zero-resource languages, which have little parallel audio-text data for
training modern supervised ASR and speech-to-text translation models, but
account for the majority of the languages spoken across the world.
