  While large language models (LMs) have shown remarkable capabilities across
numerous tasks, they often struggle with simple reasoning and planning in
physical environments, such as understanding object permanence or planning
household activities. The limitation arises from the fact that LMs are trained
only on written text and miss essential embodied knowledge and skills. In this
paper, we propose a new paradigm of enhancing LMs by finetuning them with world
models, to gain diverse embodied knowledge while retaining their general
language capabilities. Our approach deploys an embodied agent in a world model,
particularly a simulator of the physical world (VirtualHome), and acquires a
diverse set of embodied experiences through both goal-oriented planning and
random exploration. These experiences are then used to finetune LMs to teach
diverse abilities of reasoning and acting in the physical world, e.g., planning
and completing goals, object permanence and tracking, etc. Moreover, it is
desirable to preserve the generality of LMs during finetuning, which
facilitates generalizing the embodied knowledge across tasks rather than being
tied to specific simulations. We thus further introduce the classical (EWC) for
selective weight updates, combined with low-rank adapters (LoRA) for training
efficiency. Extensive experiments show our approach substantially improves base
LMs on 18 downstream tasks by 64.28% on average. In particular, the small LMs
(1.3B, 6B, and 13B) enhanced by our approach match or even outperform much
larger LMs (e.g., ChatGPT).
