  In the present study, we provided students an unfiltered access to a
state-of-the-art large language model (LLM) chatbot. The chatbot was
intentionally designed to mimic proprietary commercial chatbots such as ChatGPT
where the chatbot has not been tailored for the educational context; the
underlying engine was OpenAI GPT-4. The chatbot was integrated into online
learning materials of three courses. One of the courses focused on software
engineering with LLMs, while the two other courses were not directly related to
LLMs. Our results suggest that only a minority of students engage with the
chatbot in the courses that do not relate to LLMs. At the same time,
unsurprisingly, nearly all students in the LLM-focused course leveraged the
chatbot. In all courses, the majority of the LLM usage came from a few
superusers, whereas the majority of the students did not heavily use the
chatbot even though it was readily available and effectively provided a free
access to the OpenAI GPT-4 model. We also observe that in addition to students
using the chatbot for course-specific purposes, many use the chatbot for their
own purposes. These results suggest that the worst fears of educators -- all
students overrelying on LLMs -- did not materialize even when the chatbot
access was unfiltered. We finally discuss potential reasons for the low usage,
suggesting the need for more tailored and scaffolded LLM experiences targeted
for specific types of student use cases.
