  Generative AI tools can provide people with the ability to create virtual
environments and scenes with natural language prompts. Yet, how people will
formulate such prompts is unclear -- particularly when they inhabit the
environment that they are designing. For instance, it is likely that a person
might say, "Put a chair here", while pointing at a location. If such linguistic
features are common to people's prompts, we need to tune models to accommodate
them. In this work, we present a wizard-of-oz elicitation study with 22
participants, where we studied people's implicit expectations when verbally
prompting such programming agents to create interactive VR scenes. Our findings
show that people prompt with several implicit expectations: (1) that agents
have an embodied knowledge of the environment; (2) that agents understand
embodied prompts by users; (3) that the agents can recall previous states of
the scene and the conversation, and that (4) agents have a commonsense
understanding of objects in the scene. Further, we found that participants
prompt differently when they are prompting in situ (i.e. within the VR
environment) versus ex situ (i.e. viewing the VR environment from the outside).
To explore how our could be applied, we designed and built Oastaad, a
conversational programming agent that allows non-programmers to design
interactive VR experiences that they inhabit. Based on these explorations, we
outline new opportunities and challenges for conversational programming agents
that create VR environments.
