  In this paper, we introduce the novel concept of pedagogically aligned Large
Language Models (LLMs) that signifies a transformative shift in the application
of LLMs within educational contexts. Rather than providing direct responses to
user queries, pedagogically-aligned LLMs function as scaffolding tools,
breaking complex problems into manageable subproblems and guiding students
towards the final answer through constructive feedback and hints. The objective
is to equip learners with problem-solving strategies that deepen their
understanding and internalization of the subject matter. Previous research in
this field has primarily applied the supervised finetuning approach without
framing the objective as an alignment problem, hence not employing
reinforcement learning through human feedback (RLHF) methods. This study
reinterprets the narrative by viewing the task through the lens of alignment
and demonstrates how RLHF methods emerge naturally as a superior alternative
for aligning LLM behaviour. Building on this perspective, we propose a novel
approach for constructing a reward dataset specifically designed for the
pedagogical alignment of LLMs. We apply three state-of-the-art RLHF algorithms
and find that they outperform SFT significantly. Our qualitative analyses
across model differences and hyperparameter sensitivity further validate the
superiority of RLHF over SFT. Also, our study sheds light on the potential of
online feedback for enhancing the performance of pedagogically-aligned LLMs,
thus providing valuable insights for the advancement of these models in
educational settings.
