  Large Language Models (LLMs) handle physical commonsense information
inadequately. As a result of being trained in a disembodied setting, LLMs often
fail to predict an action's outcome in a given environment. However, predicting
the effects of an action before it is executed is crucial in planning, where
coherent sequences of actions are often needed to achieve a goal. Therefore, we
introduce the multi-modal task of predicting the outcomes of actions solely
from realistic sensory inputs (images and text). Next, we extend an LLM to
model latent representations of objects to better predict action outcomes in an
environment. We show that multi-modal models can capture physical commonsense
when augmented with visual information. Finally, we evaluate our model's
performance on novel actions and objects and find that combining modalities
help models to generalize and learn physical commonsense reasoning better.
