  Conversational tutoring systems (CTSs) offer learning experiences through
interactions based on natural language. They are recognized for promoting
cognitive engagement and improving learning outcomes, especially in reasoning
tasks. Nonetheless, the cost associated with authoring CTS content is a major
obstacle to widespread adoption and to research on effective instructional
design. In this paper, we discuss and evaluate a novel type of CTS that
leverages recent advances in large language models (LLMs) in two ways: First,
the system enables AI-assisted content authoring by inducing an easily editable
tutoring script automatically from a lesson text. Second, the system automates
the script orchestration in a learning-by-teaching format via two LLM-based
agents (Ruffle&Riley) acting as a student and a professor. The system allows
for free-form conversations that follow the ITS-typical inner and outer loop
structure. We evaluate Ruffle&Riley's ability to support biology lessons in two
between-subject online user studies (N = 200) comparing the system to simpler
QA chatbots and reading activity. Analyzing system usage patterns,
pre/post-test scores and user experience surveys, we find that Ruffle&Riley
users report high levels of engagement, understanding and perceive the offered
support as helpful. Even though Ruffle&Riley users require more time to
complete the activity, we did not find significant differences in short-term
learning gains over the reading activity. Our system architecture and user
study provide various insights for designers of future CTSs. We further
open-source our system to support ongoing research on effective instructional
design of LLM-based learning technologies.
