  Socratic questioning is an educational method that allows students to
discover answers to complex problems by asking them a series of thoughtful
questions. Generation of didactically sound questions is challenging, requiring
understanding of the reasoning process involved in the problem. We hypothesize
that such questioning strategy can not only enhance the human performance, but
also assist the math word problem (MWP) solvers. In this work, we explore the
ability of large language models (LMs) in generating sequential questions for
guiding math word problem-solving. We propose various guided question
generation schemes based on input conditioning and reinforcement learning. On
both automatic and human quality evaluations, we find that LMs constrained with
desirable question properties generate superior questions and improve the
overall performance of a math word problem solver. We conduct a preliminary
user study to examine the potential value of such question generation models in
the education domain. Results suggest that the difficulty level of problems
plays an important role in determining whether questioning improves or hinders
human performance. We discuss the future of using such questioning strategies
in education.
