  When engaging in collaborative tasks, humans efficiently exploit the semantic
structure of a conversation to optimize verbal and nonverbal interactions. But
in recent "language to code" or "language to action" models, this information
is lacking. We show how incorporating the prior discourse and nonlinguistic
context of a conversation situated in a nonlinguistic environment can improve
the "language to action" component of such interactions. We fine tune an LLM to
predict actions based on prior context; our model, NeBuLa, doubles the
net-action F1 score over the baseline on this task of Jayannavar et al.(2020).
We also investigate our model's ability to construct shapes and understand
location descriptions using a synthetic dataset.
