  Word embeddings such as ELMo have recently been shown to model word semantics
with greater efficacy through contextualized learning on large-scale language
corpora, resulting in significant improvement in state of the art across many
natural language tasks. In this work we integrate acoustic information into
contextualized lexical embeddings through the addition of multimodal inputs to
a pretrained bidirectional language model. The language model is trained on
spoken language that includes text and audio modalities. The resulting
representations from this model are multimodal and contain paralinguistic
information which can modify word meanings and provide affective information.
We show that these multimodal embeddings can be used to improve over previous
state of the art multimodal models in emotion recognition on the CMU-MOSEI
dataset.
