  In this paper, we present a novel multi-modal deep neural network
architecture that uses speech and text entanglement for learning phonetically
sound spoken-word representations. STEPs-RL is trained in a supervised manner
to predict the phonetic sequence of a target spoken-word using its contextual
spoken word's speech and text, such that the model encodes its meaningful
latent representations. Unlike existing work, we have used text along with
speech for auditory representation learning to capture semantical and
syntactical information along with the acoustic and temporal information. The
latent representations produced by our model were not only able to predict the
target phonetic sequences with an accuracy of 89.47% but were also able to
achieve competitive results to textual word representation models, Word2Vec &
FastText (trained on textual transcripts), when evaluated on four widely used
word similarity benchmark datasets. In addition, investigation of the generated
vector space also demonstrated the capability of the proposed model to capture
the phonetic structure of the spoken-words. To the best of our knowledge, none
of the existing works use speech and text entanglement for learning spoken-word
representation, which makes this work first of its kind.
