<p>My assumption is that you want to teach a professional, quality-oriented mindset about software development.</p>
<p>If you want to resemble professional software development (at least to some degree), I'd recommend a two-level approach.</p>
<p>Typically, we write code and check it locally against some test cases before we &quot;publish&quot; it. So, there's a two-step process:</p>
<ul>
<li>With the local checks, it does not matter how often we check and fail. It might take some time, but will help to improve quality.</li>
<li>If published code fails, we (our company) might get a problem, with customers complaining, or even financial losses.</li>
</ul>
<p>So, an important step in development is to decide when my code is production-ready.</p>
<p>For teaching such a mindset, a grader that penalizes failed attempts is a useful element, to be used for &quot;published&quot; code.</p>
<p>But then you'll also want to teach your students how to do the local tests, and that's a whole topic in itself: designing for testability, creating test cases that cover relevant situations, using unit testing frameworks etc.</p>
<p>So, for general CS courses, I'd refrain from penalizing failed attempts, and only do it in quality-focussed courses.</p>

CC BY-SA 4.0
2023-11-07T09:04:37.413Z
2023-11-07T09:04:37.413Z
Ralf Kleberhoff
<p>As is often the case, it depends on the goals of the course, the subject matter, the experience level of the students, the nature of the assignment and expected student outcomes.</p>
<p>But generally speaking, if you allow students infinite runs on a test suite with no penalty and allow them to see output, the risk is that they'll learn to lean too heavily on the test suite and won't learn to think through their code mentally or anticipate edge cases. These skills are important in actual development, when they're the ones writing the test suite and need to imagine how the software will be used to anticipate as many failure states and usability issues as possible.</p>
<p>On the other hand, if you pass restrictions on submissions, it can be discouraging to students, especially those first starting out. If the written specification isn't clear, students may feel that you're unfairly throwing obstacles in their way. In an extreme case, where no test harness is provided, even the basic header of the function can be frustrating to write. Providing basic tests ensures the input and output formats are respected, allowing the student to move on to the implementation logic.</p>
<p>The parameters or knobs you have for autograding are typically (beyond penalizing repeated submissions mentioned in the question):</p>
<ul>
<li>Do you provide test cases used for final grading up front?</li>
<li>Do you hide or show test case input to the student when they submit their code?</li>
<li>Do you hide or show student code output sent to the grading suite?</li>
<li>Do you hide or show correctness for submitted solutions?</li>
<li>Do you limit the number or frequency of submissions?</li>
<li>Do you penalize submissions beyond a limit?</li>
</ul>
<p>In all cases, you can answer with &quot;none, some, or all&quot;. If &quot;some&quot;, then you can choose the extent.</p>
<p>A reasonable common-case approach might be:</p>
<ul>
<li>Provide some small test cases to students in full to get them started</li>
<li>Students can run the sample tests as many times as they want</li>
<li>Students can see their input and output on sample tests</li>
<li>Encourage students to supplement the sample tests with their own tests</li>
<li>Reserve some test cases as &quot;hidden&quot; submission-only tests which typically cover edge cases and performance concerns that push the code to the limits of the written specification</li>
<li>Possibly show output on hidden tests (keeping in mind that this can be abused to reveal inputs as well)</li>
<li>Limit submissions to some reasonable amount, say, 25, to prevent brute-forcing the autograder</li>
<li>Don't penalize repeated submissions, or, if you do, use a very generous threshold.</li>
</ul>
<p>As always, be clear and consistent about how you're setting these knobs, both to yourself and to your students. Prepare to adjust the knobs after each cohort.</p>
<p>Regardless of what you feel is appropriate, communicate how tests and grading work, then stick to it on each assignment. Start with an easy assignment to get students into the grading flow, then ramp up difficulty.</p>
<p>If you do withhold information, or penalize or limit repeated attempts, explain why.</p>

CC BY-SA 4.0
2023-11-07T17:12:42.850Z
2023-11-07T17:28:39.717Z
2023-11-07T17:28:39.717Z
ggorlen
<p>I believe there is a case for limiting the number of submissions. As the feedback from each test cases shows the input, their output and the correct output, the students have all the information needed to run their own tests. They can change their code, try the input and see how it compares to the correct output. I don't believe it is useful to simply make a change and throw it at the autograder hoping the results are correct. A student is not learning debugging skills if they simply rely on the autograder.</p>
<p>To do this, the instructor may specify two values: the time between submissions, and the total submissions over the life of the assignment. The instructor may choose to set neither and then the students have unlimited tries, throttled only by the turnaround of the autograder.</p>

CC BY-SA 4.0
2023-11-07T18:05:55.997Z
2023-11-07T18:05:55.997Z
Fritz Sieker
<p>I used to think about a similar setup in my earlier years of teaching CS, but now I'm about 15 years in, and I don't see the benefits.</p>
<p>I'm trying to imagine a work environment in which intermediate work should be hidden from view, and the only thing I can think of is poisonous workplaces where either fear of job security or fear of humiliation prevents employees from truly collaborating.  Certainly, this would not lead to smarter work.</p>
<p>Within a school situation, I can, perhaps, imagine a benefit to assessments, but they are already typically time limited.</p>
<p>Beyond assessments, I simply cannot imagine a learning benefit that students would obtain from such restrictions.  I can guess is that it might be a misguided attempt to get students to code more thoughtfully, and code less by brownian motion.  But I don't believe that it will achieve those ends.  At best, it will cause students to code off to the side, and then paste in their code when they believe it is complete.  This increases their setup work, and gives them an extra task at the end, but those tasks do not forward any particular learning goals.</p>
<p>I would steer clear.</p>

CC BY-SA 4.0
2023-11-06T18:29:42.043Z
2023-11-06T18:29:42.043Z
Ben I.
<p>In a computer science course, is it a good idea to create automatically graded problems that allow students only a couple of attempts and give fewer points if he or she answers many times? Namely, it will give some experience that in work life, do not publish unfinished code. But it would make people shy to show their efforts.</p>

CC BY-SA 4.0
2023-11-06T16:01:59.177Z
Interesting idea, but give a bit more context. It will improve any answers you get.
Buffy
<auto-graders>
Should a student be allowed to try to answer a question only a few times?
# Should a student be allowed to try to answer a question only a few times?

(Asked by: username_0 on 2023-11-06)

In a computer science course, is it a good idea to create automatically graded problems that allow students only a couple of attempts and give fewer points if he or she answers many times? Namely, it will give some experience that in work life, do not publish unfinished code. But it would make people shy to show their efforts.

**Tags:** auto-graders

> **Comments:**
> **username_1**: Interesting idea, but give a bit more context. It will improve any answers you get.  

---

## 4 answers

### Answer by username_5 on 2023-11-06. Score: 4 | Accepted answer

I used to think about a similar setup in my earlier years of teaching CS, but now I'm about 15 years in, and I don't see the benefits.

I'm trying to imagine a work environment in which intermediate work should be hidden from view, and the only thing I can think of is poisonous workplaces where either fear of job security or fear of humiliation prevents employees from truly collaborating. Certainly, this would not lead to smarter work.

Within a school situation, I can, perhaps, imagine a benefit to assessments, but they are already typically time limited.

Beyond assessments, I simply cannot imagine a learning benefit that students would obtain from such restrictions. I can guess is that it might be a misguided attempt to get students to code more thoughtfully, and code less by brownian motion. But I don't believe that it will achieve those ends. At best, it will cause students to code off to the side, and then paste in their code when they believe it is complete. This increases their setup work, and gives them an extra task at the end, but those tasks do not forward any particular learning goals.

I would steer clear.


---

### Answer by username_2 on 2023-11-07. Score: 4

My assumption is that you want to teach a professional, quality-oriented mindset about software development.

If you want to resemble professional software development (at least to some degree), I'd recommend a two-level approach.

Typically, we write code and check it locally against some test cases before we "publish" it. So, there's a two-step process:

* With the local checks, it does not matter how often we check and fail. It might take some time, but will help to improve quality.
* If published code fails, we (our company) might get a problem, with customers complaining, or even financial losses.

So, an important step in development is to decide when my code is production-ready.

For teaching such a mindset, a grader that penalizes failed attempts is a useful element, to be used for "published" code.

But then you'll also want to teach your students how to do the local tests, and that's a whole topic in itself: designing for testability, creating test cases that cover relevant situations, using unit testing frameworks etc.

So, for general CS courses, I'd refrain from penalizing failed attempts, and only do it in quality-focussed courses.


---

### Answer by username_3 on 2023-11-07. Score: 1

As is often the case, it depends on the goals of the course, the subject matter, the experience level of the students, the nature of the assignment and expected student outcomes.

But generally speaking, if you allow students infinite runs on a test suite with no penalty and allow them to see output, the risk is that they'll learn to lean too heavily on the test suite and won't learn to think through their code mentally or anticipate edge cases. These skills are important in actual development, when they're the ones writing the test suite and need to imagine how the software will be used to anticipate as many failure states and usability issues as possible.

On the other hand, if you pass restrictions on submissions, it can be discouraging to students, especially those first starting out. If the written specification isn't clear, students may feel that you're unfairly throwing obstacles in their way. In an extreme case, where no test harness is provided, even the basic header of the function can be frustrating to write. Providing basic tests ensures the input and output formats are respected, allowing the student to move on to the implementation logic.

The parameters or knobs you have for autograding are typically (beyond penalizing repeated submissions mentioned in the question):

* Do you provide test cases used for final grading up front?
* Do you hide or show test case input to the student when they submit their code?
* Do you hide or show student code output sent to the grading suite?
* Do you hide or show correctness for submitted solutions?
* Do you limit the number or frequency of submissions?
* Do you penalize submissions beyond a limit?

In all cases, you can answer with "none, some, or all". If "some", then you can choose the extent.

A reasonable common-case approach might be:

* Provide some small test cases to students in full to get them started
* Students can run the sample tests as many times as they want
* Students can see their input and output on sample tests
* Encourage students to supplement the sample tests with their own tests
* Reserve some test cases as "hidden" submission-only tests which typically cover edge cases and performance concerns that push the code to the limits of the written specification
* Possibly show output on hidden tests (keeping in mind that this can be abused to reveal inputs as well)
* Limit submissions to some reasonable amount, say, 25, to prevent brute-forcing the autograder
* Don't penalize repeated submissions, or, if you do, use a very generous threshold.

As always, be clear and consistent about how you're setting these knobs, both to yourself and to your students. Prepare to adjust the knobs after each cohort.

Regardless of what you feel is appropriate, communicate how tests and grading work, then stick to it on each assignment. Start with an easy assignment to get students into the grading flow, then ramp up difficulty.

If you do withhold information, or penalize or limit repeated attempts, explain why.


---

### Answer by username_4 on 2023-11-07. Score: 1

I believe there is a case for limiting the number of submissions. As the feedback from each test cases shows the input, their output and the correct output, the students have all the information needed to run their own tests. They can change their code, try the input and see how it compares to the correct output. I don't believe it is useful to simply make a change and throw it at the autograder hoping the results are correct. A student is not learning debugging skills if they simply rely on the autograder.

To do this, the instructor may specify two values: the time between submissions, and the total submissions over the life of the assignment. The instructor may choose to set neither and then the students have unlimited tries, throttled only by the turnaround of the autograder.


---

