  Chatbot interfaces for LLMs enable students to get immediate, interactive
help on homework assignments, but even a thoughtfully-designed bot may not
serve all pedagogical goals. In this paper, we report on the development and
deployment of a GPT-4-based interactive homework assistant ("61A-Bot") for
students in a large CS1 course; over 2000 students made over 100,000 requests
of our bot across two semesters. Our assistant offers one-shot, contextual
feedback, through both a "Get Help" button within a popular code editor, as
well as a "get feedback" feature within our command-line autograder. These
triggers wrap student code in a custom prompt that supports our pedagogical
goals and avoids providing solutions directly. We discuss our development
process and deployment, then analyze possible impacts of our Bot on students,
primarily through student feedback and how long it takes students to complete
homework problems. We ask: how does access to 61A-Bot impact homework
completion time and subsequent course performance? In addition to reductions in
homework-related question rates in our course forum, we find substantial
reductions in homework completion time. These are most pronounced for students
in the 50th-80th percentile, with reductions of over 30 minutes, over 4
standard deviations faster than the mean in prior semesters. However, it is not
clear that these effects transfer to assignment contexts where the Bot is not
available: we observe speedups in some contexts, no change in others, and some
assignments later in the semester even show a slowdown instead. Though we have
begun to disentangle these effects, further research is needed.
