  Providing personalized assistance at scale is a long-standing challenge for
computing educators, but a new generation of tools powered by large language
models (LLMs) offers immense promise. Such tools can, in theory, provide
on-demand help in large class settings and be configured with appropriate
guardrails to prevent misuse and mitigate common concerns around learner
over-reliance. However, the deployment of LLM-powered tools in authentic
classroom settings is still rare, and very little is currently known about how
students will use them in practice and what type of help they will seek. To
address this, we examine students' use of an innovative LLM-powered tool that
provides on-demand programming assistance without revealing solutions directly.
We deployed the tool for 12 weeks in an introductory computer and data science
course ($n = 52$), collecting more than 2,500 queries submitted by students
throughout the term. We manually categorized all student queries based on the
type of assistance sought, and we automatically analyzed several additional
query characteristics. We found that most queries requested immediate help with
programming assignments, whereas fewer requests asked for help on related
concepts or for deepening conceptual understanding. Furthermore, students often
provided minimal information to the tool, suggesting this is an area in which
targeted instruction would be beneficial. We also found that students who
achieved more success in the course tended to have used the tool more
frequently overall. Lessons from this research can be leveraged by programming
educators and institutions who plan to augment their teaching with emerging
LLM-powered tools.
