  Large language models (LLMs) have recently been applied in software
engineering to perform tasks such as translating code between programming
languages, generating code from natural language, and autocompleting code as it
is being written. When used within development tools, these systems typically
treat each model invocation independently from all previous invocations, and
only a specific limited functionality is exposed within the user interface.
This approach to user interaction misses an opportunity for users to more
deeply engage with the model by having the context of their previous
interactions, as well as the context of their code, inform the model's
responses. We developed a prototype system -- the Programmer's Assistant -- in
order to explore the utility of conversational interactions grounded in code,
as well as software engineers' receptiveness to the idea of conversing with,
rather than invoking, a code-fluent LLM. Through an evaluation with 42
participants with varied levels of programming experience, we found that our
system was capable of conducting extended, multi-turn discussions, and that it
enabled additional knowledge and capabilities beyond code generation to emerge
from the LLM. Despite skeptical initial expectations for conversational
programming assistance, participants were impressed by the breadth of the
assistant's capabilities, the quality of its responses, and its potential for
improving their productivity. Our work demonstrates the unique potential of
conversational interactions with LLMs for co-creative processes like software
development.
