  Advances in natural language processing have resulted in large language
models (LLMs) that are capable of generating understandable and sensible
written text. Recent versions of these models, such as OpenAI Codex and GPT-3,
can generate code and code explanations. However, it is unclear whether and how
students might engage with such explanations. In this paper, we report on our
experiences generating multiple code explanation types using LLMs and
integrating them into an interactive e-book on web software development. We
modified the e-book to make LLM-generated code explanations accessible through
buttons next to code snippets in the materials, which allowed us to track the
use of the explanations as well as to ask for feedback on their utility. Three
different types of explanations were available for students for each
explainable code snippet; a line-by-line explanation, a list of important
concepts, and a high-level summary of the code. Our preliminary results show
that all varieties of explanations were viewed by students and that the
majority of students perceived the code explanations as helpful to them.
However, student engagement appeared to vary by code snippet complexity,
explanation type, and code snippet length. Drawing on our experiences, we
discuss future directions for integrating explanations generated by LLMs into
existing computer science classrooms.
