  In this paper, we explore the unsupervised learning of a semantic embedding
space for co-occurring sensory inputs. Specifically, we focus on the task of
learning a semantic vector space for both spoken and handwritten digits using
the TIDIGITs and MNIST datasets. Current techniques encode image and
audio/textual inputs directly to semantic embeddings. In contrast, our
technique maps an input to the mean and log variance vectors of a diagonal
Gaussian from which sample semantic embeddings are drawn. In addition to
encouraging semantic similarity between co-occurring inputs,our loss function
includes a regularization term borrowed from variational autoencoders (VAEs)
which drives the posterior distributions over embeddings to be unit Gaussian.
We can use this regularization term to filter out modality information while
preserving semantic information. We speculate this technique may be more
broadly applicable to other areas of cross-modality/domain information
retrieval and transfer learning.
