  In this paper, we extend a symbolic association framework for being able to
handle missing elements in multimodal sequences. The general scope of the work
is the symbolic associations of object-word mappings as it happens in language
development in infants. In other words, two different representations of the
same abstract concepts can associate in both directions. This scenario has been
long interested in Artificial Intelligence, Psychology, and Neuroscience. In
this work, we extend a recent approach for multimodal sequences (visual and
audio) to also cope with missing elements in one or both modalities. Our method
uses two parallel Long Short-Term Memories (LSTMs) with a learning rule based
on EM-algorithm. It aligns both LSTM outputs via Dynamic Time Warping (DTW). We
propose to include an extra step for the combination with the max operation for
exploiting the common elements between both sequences. The motivation behind is
that the combination acts as a condition selector for choosing the best
representation from both LSTMs. We evaluated the proposed extension in the
following scenarios: missing elements in one modality (visual or audio) and
missing elements in both modalities (visual and sound). The performance of our
extension reaches better results than the original model and similar results to
individual LSTM trained in each modality.
