  In feedback generation for logical errors in programming assignments, large
language model (LLM)-based methods have shown great promise. These methods ask
the LLM to generate feedback given the problem statement and a student's
(buggy) submission. There are several issues with these types of methods.
First, the generated feedback messages are often too direct in revealing the
error in the submission and thus diminish valuable opportunities for the
student to learn. Second, they do not consider the student's learning context,
i.e., their previous submissions, current knowledge, etc. Third, they are not
layered since existing methods use a single, shared prompt for all student
submissions. In this paper, we explore using LLMs to generate a
"feedback-ladder", i.e., multiple levels of feedback for the same
problem-submission pair. We evaluate the quality of the generated
feedback-ladder via a user study with students, educators, and researchers. We
have observed diminishing effectiveness for higher-level feedback and
higher-scoring submissions overall in the study. In practice, our method
enables teachers to select an appropriate level of feedback to show to a
student based on their personal learning context, or in a progressive manner to
go more detailed if a higher-level feedback fails to correct the student's
error.
