  Human beings possess the capability to multiply a melange of multisensory
cues while actively exploring and interacting with the 3D world. Current
multi-modal large language models, however, passively absorb sensory data as
inputs, lacking the capacity to actively interact with the objects in the 3D
environment and dynamically collect their multisensory information. To usher in
the study of this area, we propose MultiPLY, a multisensory embodied large
language model that could incorporate multisensory interactive data, including
visual, audio, tactile, and thermal information into large language models,
thereby establishing the correlation among words, actions, and percepts. To
this end, we first collect Multisensory Universe, a large-scale multisensory
interaction dataset comprising 500k data by deploying an LLM-powered embodied
agent to engage with the 3D environment. To perform instruction tuning with
pre-trained LLM on such generated data, we first encode the 3D scene as
abstracted object-centric representations and then introduce action tokens
denoting that the embodied agent takes certain actions within the environment,
as well as state tokens that represent the multisensory state observations of
the agent at each time step. In the inference time, MultiPLY could generate
action tokens, instructing the agent to take the action in the environment and
obtain the next multisensory state observation. The observation is then
appended back to the LLM via state tokens to generate subsequent text or action
tokens. We demonstrate that MultiPLY outperforms baselines by a large margin
through a diverse set of embodied tasks involving object retrieval, tool use,
multisensory captioning, and task decomposition.
