  Large language models (LLMs) are flexible, personalizable, and available,
which makes their use within Intelligent Tutoring Systems (ITSs) appealing.
However, that flexibility creates risks: inaccuracies, harmful content, and
non-curricular material. Ethically deploying LLM-backed ITS systems requires
designing safeguards that ensure positive experiences for students. We describe
the design of a conversational system integrated into an ITS, and our
experience evaluating its safety with red-teaming, an in-classroom usability
test, and field deployment. We present empirical data from more than 8,000
student conversations with this system, finding that GPT-3.5 rarely generates
inappropriate messages. Comparatively more common is inappropriate messages
from students, which prompts us to reason about safeguarding as a content
moderation and classroom management problem. The student interaction behaviors
we observe provide implications for designers - to focus on student inputs as a
content moderation problem - and implications for researchers - to focus on
subtle forms of bad content.
