  This work addresses the problem of sensing the world: how to learn a
multimodal representation of a reinforcement learning agent's environment that
allows the execution of tasks under incomplete perceptual conditions. To
address such problem, we argue for hierarchy in the design of representation
models and contribute with a novel multimodal representation model, MUSE. The
proposed model learns hierarchical representations: low-level modality-specific
representations, encoded from raw observation data, and a high-level multimodal
representation, encoding joint-modality information to allow robust state
estimation. We employ MUSE as the sensory representation model of deep
reinforcement learning agents provided with multimodal observations in Atari
games. We perform a comparative study over different designs of reinforcement
learning agents, showing that MUSE allows agents to perform tasks under
incomplete perceptual experience with minimal performance loss. Finally, we
evaluate the performance of MUSE in literature-standard multimodal scenarios
with higher number and more complex modalities, showing that it outperforms
state-of-the-art multimodal variational autoencoders in single and
cross-modality generation.
