<p>Our university has a public grade distribution database, so I did a really quick analysis on some historical data to see if there was any support for this idea. To my surprise, it appears that there might be. But, my analysis is very limited.</p>

<p>I downloaded the grade distributions for all intro psychology and intro computer science classes from 2010-2014. This gave me the grades of about 15,000 psychology students and about 1,500 computer science students. I summed up the count for each letter grade, and then normalized to a percentage of students. Apologies for the somewhat ugly chart, but here's what I found:</p>

<p><img src="https://i.sstatic.net/pKOVu.png" alt="enter image description here"></p>

<p>The distribution for letter grades seems pretty consistent across both disciplines, but what jumps out to me is the assignment of "W" grades, which is the grade received when a student withdraws from the course after being enrolled for at least the first week of classes. About 20% of all intro CS students withdraw from the class, while less than 5% of intro psych students do. This might indicate that a larger number of CS students find the course to be too challenging for them, and withdraw rather than get a low grade.</p>

CC BY-SA 3.0
2017-12-16T01:47:10.353Z
Do you (still) have the data in numerical format somewhere? I'd hate to have to reverse it from the graph...
got trolled too much this week
CC BY-SA 3.0
2015-03-11T14:03:15.070Z
2015-03-11T17:21:07.407Z
2015-03-11T17:21:07.407Z
Josh
<p>(This is a bit brief given the complexity of the question. I'll expand later if I can summarize more of the findings.)</p>

<p>Probably the only thing which is certain is the high-failure rate in CS1-level classes, worldwide. From a <a href="https://doi.org/10.1145/2591708.2591749" rel="nofollow noreferrer">recent (2014) meta-analysis</a>:</p>

<blockquote>
  <p>In this paper, we answer the call for further substantial evidence on the CS1 failure rate phenomenon, by performing a systematic review of introductory programming literature, and a statistical analysis on pass rate data extracted from relevant articles. Pass rates describing the outcomes of 161 CS1 courses that ran in 15 different countries, across 51 institutions were extracted and analysed. An almost identical mean worldwide pass rate of 67.7% was found. Moderator analysis revealed significant, but perhaps not substantial differences in pass rates based upon: grade level, country, and class size. However, pass rates were found not to have significantly differed over time, or based upon the programming language taught in the course.</p>
</blockquote>

<p>There was a prior world-wide study in 2007 (using online self-reported data) which agreed with this number (~33% failure rate), although the distribution observed for the passing students was different:</p>

<p><a href="https://i.sstatic.net/Dx81Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Dx81Y.png" alt="enter image description here"></a></p>

<p>Alas neither paper compares this number with other fields, but the <a href="https://doi.org/10.1145/1272848.1272879" rel="nofollow noreferrer">2007 one</a> also calculated that graduation from CS tertiary education is low worldwide, 26.8% so getting past CS1 hardly guarantees smooth sailing thereafter.</p>

<p><a href="https://i.sstatic.net/ARtUj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ARtUj.png" alt="enter image description here"></a></p>

<p>The latter [counterproductively 3D] graph is based on UNESCO data, which is why North America is absent (the US boycott[ed] UNESCO.)</p>

<p>There is however <a href="https://doi.org/10.1145/1026487.1008029" rel="nofollow noreferrer">one paper</a> which does state that:</p>

<blockquote>
  <p>Studies in Ireland [1] have shown that Computer Science has the highest
  failure rate, giving a 26.9% non-completion rate for Computing
  courses.</p>
  
  <p>[... citing:]</p>
  
  <p>[1] Mark Morgan, Rita Flanagan and Thomas Kellaghan, "Higher Education Report A Study of Non-Completion in Undergraduate University Courses" published February 2001</p>
</blockquote>

<p>Several theories for the CS1 failure rate have been advanced: genetic basis,  <a href="http://dx.doi.org/10.1080/08993401003612167" rel="nofollow noreferrer">"learning edge momentum"</a>, and/or <a href="https://doi.org/10.1145/2493394.2493416" rel="nofollow noreferrer">stumbling blocks</a> but it's too early to tell if any of these have convincing empirical support. (In fact the last paper argues that insofar there isn't a good data gathering strategy that would even allow one to one to decide.) And <a href="https://cte-blog.uwaterloo.ca/?p=5153" rel="nofollow noreferrer">one academic blog</a> suggests (based on its own data) that none might be correct. <a href="http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1010&amp;context=sais2015" rel="nofollow noreferrer">Yet another paper</a> suggests that (Duckworth's) grit plays a substantial role. And with respect to the final grade being an aggregation <a href="https://dl.acm.org/citation.cfm?id=2459943" rel="nofollow noreferrer">one paper</a> (based on a single course) suggests that students' bimodal final grades come from the practical side of the evaluation:</p>

<p><a href="https://i.sstatic.net/LejvW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LejvW.png" alt="enter image description here"></a></p>

<p>Alas I was not able to find other papers looking into this aspect, however it does look to me that the grit paper does point in the same direction; it found that grit was a much better predictor than ACT (itself considered a proxy for IQ) for CS1 success.</p>

<p>The humps paper mentiond in the question is not taken very seriously (most of these papers dont' cite it), and <a href="http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-two-humps-programming-aptitude-test-canned-for-overzealous-conclusion/" rel="nofollow noreferrer">even by its own authors themselves these days</a>. It also does not seem to have been the first to observe the grade distribution phenomenon. I'm not entirely sure why it became popular on-line. Its strong language and/or the SE-effect maybe. For instance <a href="https://doi.org/10.1145/1083431.1083474" rel="nofollow noreferrer">this 2005 paper</a> noted the 30% to 40% failure rate as well.</p>

<hr>

<p>Also here's a graph with the failure rate by country from the 2014 meta-analysis:</p>

<p><a href="https://i.sstatic.net/xDyKL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xDyKL.png" alt="enter image description here"></a></p>

<p>The authors note that the top 3 failure leaders Portugal, Germany and Brazil were represented by a pretty small sample in this study, so the data for those countries cannot really be relied upon. On the other hand, "of the 4 countries
which made 80% of the sample, Finland was found to have the lowest pass rate 57.7%". The other three countries in this 80% bulk of the data, US, UK, and Australia had statistically indistinguishable rates.</p>

<p>Also, the methodology should have been more upfront in this meta-analysis but they do say "Currently we have found in this study 3/10 students do not complete, or fail CS1." So they definitely count withdrawals as failure, basically, for the statics they reported.</p>

CC BY-SA 3.0
2017-12-15T12:26:20.620Z
2017-12-16T02:41:08.177Z
2017-12-16T02:41:08.177Z
got trolled too much this week
<p>According to the research of Elizabeth Patitsas, <a href="https://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html" rel="nofollow noreferrer">the distribution is not bimodal</a>. She sampled 778 University of British Columbia CS courses, as well as 5 University of Toronto CS courses and found the vast majority to not be bimodal:</p>

<blockquote>
  <p>An estimated 85.1% of the final grades in UBC's undergrad CS classes
  are normally-distributed. 5.8% of the classes tested as being bimodal,
  which isn't a whole lot more than the false positive rate I'd expect
  to see (5%).</p>
</blockquote>

<p>You can read more about these results in the blog post linked above or in "<a href="https://dl.acm.org/citation.cfm?doid=2960310.2960312" rel="nofollow noreferrer">Evidence That Computer Science Grades Are Not Bimodal</a>" by Elizabeth Patitsas, Jesse Berlin, Michelle Craig and Steve Easterbrook.</p>

<p>To make things worse, when discussing this, <a href="https://computinged.wordpress.com/2016/04/13/where-cs-teachers-see-bimodality-guest-blog-post-from-elizabeth-patitsas/" rel="nofollow noreferrer">people who believe the distribution is bimodal are more likely to see a bimodal distribution in a normally distributed sampling</a>:</p>

<blockquote>
  <p>We delayed the debriefing until after our analysis was complete, on
  the assumption you’d want to know the preliminary results of the
  study. We indeed found that participants who agreed more strongly with
  the statement “Some students are innately predisposed to do better at
  CS than others” were statistically significantly more likely to label
  ambiguous distributions as bimodal.</p>
</blockquote>

CC BY-SA 3.0
2017-12-16T00:44:16.117Z
Nice find; the psychology of the observer explained (in the paper abstract):"A
random half of participants were primed to think about the
fact that CS grades are commonly thought to be bimodal;
these participants were more likely to label ambiguous dis-
tributions as \bimodal". Participants were also more likely
to label distributions as bimodal if they believed that some
students are innately predisposed to do better at CS. These
results suggest that bimodal grades are instructional folklore
in CS, caused by conrmation bias and instructor beliefs
about their students."
got trolled too much this week
CC BY-SA 3.0
2017-12-16T01:29:30.110Z
I do have to wonder if the UBC data is representative though. They do mention a paper with an over-lapping set of authors (Zingaro et al.) in which the exam is turned into many fine-grained questions. So, if this practice is commonly applied at UBC (but not in other places), then UBC data is not representative. Ah, never mind the UBC data is from 1996 to 2013, while the Zingaro paper is from 2012, so not much chance it was applied unless it was "prior art" that nobody thought to publish about for years (unlikely).
got trolled too much this week
CC BY-SA 3.0
2017-12-16T01:31:29.470Z
@Fizz fair criticism which she acknowledges in her blog post. I guess we could do the same tests on Josh's data?
Seanny123
CC BY-SA 3.0
2017-12-16T01:39:09.217Z
Good point on trying that with different data. The methodological question is what do we do with the large number of withdrawals. The Patitsas paper says NOTHING about that... which is probably the biggest question mark, now that I realized it.
got trolled too much this week
CC BY-SA 3.0
2017-12-16T02:24:48.827Z
I'm pretty sure the 2014 meta-analysis counted withdrawals as failed. That's because they have ~30% average failure for US (it has a graph by country as well), so with Josh's data, you'd definitely have to add F+W to get anywhere near that 30% number. Also, at some US institution I'm familiar with, the D's would be failure as well because the students need at least a C to enroll in subsequent CS classes. The Canadian failure rate in the meta-analysis was only 20%, by the way.
got trolled too much this week
CC BY-SA 3.0
2017-12-16T02:39:39.717Z
Yeah, the methodology should have been more upfront in the meta-analysis but they do say " Currently we have found in this study
3/10 students do not complete, or fail CS1." So they definitely count withdrawals as failure, basically.
got trolled too much this week
CC BY-SA 3.0
2017-12-15T22:04:14.403Z
2017-12-15T22:04:14.403Z
Seanny123
<p>On Programmers SE there is a popular question about scientific evidence for <a href="https://softwareengineering.stackexchange.com/questions/163631/has-not-everyone-can-be-a-programmer-been-studied">whether some people either are or are not able to program</a>. </p>

<p><a href="https://softwareengineering.stackexchange.com/a/163633/4488">One person cites</a> an unpublished manuscript by Dehnadi and Bornat (2006) who state in an abstract:</p>

<blockquote>
  <p>All teachers of programming find that their results display a 'double
  hump'. It is as if there are two populations: those who can, and those
  who cannot, each with its own independent bell curve. </p>
</blockquote>

<p>That said, educators have choice in how individual assessments are scored, how assessments are combined, and whether and how any final scores are scaled. These could all influence the shape of the final grade distribution.</p>

<p>I also imagine that to get a double hump would suggest that the variables that best predict student performance (e.g., time spent on subject, general ability, subject-specific ability) would have to either show a double hump or show a discontinuous relationship with performance. In particular, I can see that students that drop out of a subject without formally un-enrolling are likely to fail very badly and appear in a separate hump. I also suspect that certain types of subjects where answers are clearly correct or incorrect and where intuition from everyday life is less directly relevant may also give rise to a process where it appears as if people can either do the subject or they can't.</p>

<h3>Questions</h3>

<ul>
<li>What is the distribution of grades in university subjects?</li>
<li>To what extent does the distribution of grades in programming subjects differ from non-programming subjects? Is it bimodal?</li>
<li>Is there any theories or empirical evidence about what leads to bimodal grade distributions?</li>
</ul>

<h3>References</h3>

<ul>
<li>Dehnadi, S. &amp; Bornat, R. (2006). The camel has two humps (working title). Middlesex University, UK. <a href="http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf" rel="nofollow noreferrer">PDF</a> and <a href="http://www.eis.mdx.ac.uk/research/PhDArea/saeed/" rel="nofollow noreferrer">summary page</a>.</li>
</ul>

CC BY-SA 3.0
2017-12-16T01:21:43.797Z
It turns out the question is also being discussed on CSE: https://cseducators.stackexchange.com/questions/756/do-you-see-the-bimodal-distribution-too
got trolled too much this week
<intelligence><performance><educational-psychology>
What is the distribution of performance in academic programming subjects?
Jeromy Anglim
# What is the distribution of performance in academic programming subjects?

(Asked by: username_0 on 2012-09-05)

On Programmers SE there is a popular question about scientific evidence for [whether some people either are or are not able to program](https://softwareengineering.stackexchange.com/questions/163631/has-not-everyone-can-be-a-programmer-been-studied).

[One person cites](https://softwareengineering.stackexchange.com/a/163633/4488) an unpublished manuscript by Dehnadi and Bornat (2006) who state in an abstract:

> All teachers of programming find that their results display a 'double
> hump'. It is as if there are two populations: those who can, and those
> who cannot, each with its own independent bell curve.

That said, educators have choice in how individual assessments are scored, how assessments are combined, and whether and how any final scores are scaled. These could all influence the shape of the final grade distribution.

I also imagine that to get a double hump would suggest that the variables that best predict student performance (e.g., time spent on subject, general ability, subject-specific ability) would have to either show a double hump or show a discontinuous relationship with performance. In particular, I can see that students that drop out of a subject without formally un-enrolling are likely to fail very badly and appear in a separate hump. I also suspect that certain types of subjects where answers are clearly correct or incorrect and where intuition from everyday life is less directly relevant may also give rise to a process where it appears as if people can either do the subject or they can't.

### Questions

* What is the distribution of grades in university subjects?
* To what extent does the distribution of grades in programming subjects differ from non-programming subjects? Is it bimodal?
* Is there any theories or empirical evidence about what leads to bimodal grade distributions?

### References

* Dehnadi, S. & Bornat, R. (2006). The camel has two humps (working title). Middlesex University, UK. [PDF](http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf) and [summary page](http://www.eis.mdx.ac.uk/research/PhDArea/saeed/).

**Tags:** intelligence, performance, educational-psychology

> **Comments:**
> **username_1**: It turns out the question is also being discussed on CSE: https://cseducators.stackexchange.com/questions/756/do-you-see-the-bimodal-distribution-too  

---

## 3 answers

### Answer by username_2 on 2015-03-11. Score: 5

Our university has a public grade distribution database, so I did a really quick analysis on some historical data to see if there was any support for this idea. To my surprise, it appears that there might be. But, my analysis is very limited.

I downloaded the grade distributions for all intro psychology and intro computer science classes from 2010-2014. This gave me the grades of about 15,000 psychology students and about 1,500 computer science students. I summed up the count for each letter grade, and then normalized to a percentage of students. Apologies for the somewhat ugly chart, but here's what I found:

![enter image description here](https://i.sstatic.net/pKOVu.png)

The distribution for letter grades seems pretty consistent across both disciplines, but what jumps out to me is the assignment of "W" grades, which is the grade received when a student withdraws from the course after being enrolled for at least the first week of classes. About 20% of all intro CS students withdraw from the class, while less than 5% of intro psych students do. This might indicate that a larger number of CS students find the course to be too challenging for them, and withdraw rather than get a low grade.

> **Comments:**
> **username_1**: Do you (still) have the data in numerical format somewhere? I'd hate to have to reverse it from the graph...  

---

### Answer by username_1 on 2017-12-15. Score: 2

(This is a bit brief given the complexity of the question. I'll expand later if I can summarize more of the findings.)

Probably the only thing which is certain is the high-failure rate in CS1-level classes, worldwide. From a [recent (2014) meta-analysis](https://doi.org/10.1145/2591708.2591749):

> In this paper, we answer the call for further substantial evidence on the CS1 failure rate phenomenon, by performing a systematic review of introductory programming literature, and a statistical analysis on pass rate data extracted from relevant articles. Pass rates describing the outcomes of 161 CS1 courses that ran in 15 different countries, across 51 institutions were extracted and analysed. An almost identical mean worldwide pass rate of 67.7% was found. Moderator analysis revealed significant, but perhaps not substantial differences in pass rates based upon: grade level, country, and class size. However, pass rates were found not to have significantly differed over time, or based upon the programming language taught in the course.

There was a prior world-wide study in 2007 (using online self-reported data) which agreed with this number (~33% failure rate), although the distribution observed for the passing students was different:

[![enter image description here](https://i.sstatic.net/Dx81Y.png)](https://i.sstatic.net/Dx81Y.png)

Alas neither paper compares this number with other fields, but the [2007 one](https://doi.org/10.1145/1272848.1272879) also calculated that graduation from CS tertiary education is low worldwide, 26.8% so getting past CS1 hardly guarantees smooth sailing thereafter.

[![enter image description here](https://i.sstatic.net/ARtUj.png)](https://i.sstatic.net/ARtUj.png)

The latter [counterproductively 3D] graph is based on UNESCO data, which is why North America is absent (the US boycott[ed] UNESCO.)

There is however [one paper](https://doi.org/10.1145/1026487.1008029) which does state that:

> Studies in Ireland [1] have shown that Computer Science has the highest
> failure rate, giving a 26.9% non-completion rate for Computing
> courses.
>
> [... citing:]
>
> [1] Mark Morgan, Rita Flanagan and Thomas Kellaghan, "Higher Education Report A Study of Non-Completion in Undergraduate University Courses" published February 2001

Several theories for the CS1 failure rate have been advanced: genetic basis, ["learning edge momentum"](http://dx.doi.org/10.1080/08993401003612167), and/or [stumbling blocks](https://doi.org/10.1145/2493394.2493416) but it's too early to tell if any of these have convincing empirical support. (In fact the last paper argues that insofar there isn't a good data gathering strategy that would even allow one to one to decide.) And [one academic blog](https://cte-blog.uwaterloo.ca/?p=5153) suggests (based on its own data) that none might be correct. [Yet another paper](http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1010&context=sais2015) suggests that (Duckworth's) grit plays a substantial role. And with respect to the final grade being an aggregation [one paper](https://dl.acm.org/citation.cfm?id=2459943) (based on a single course) suggests that students' bimodal final grades come from the practical side of the evaluation:

[![enter image description here](https://i.sstatic.net/LejvW.png)](https://i.sstatic.net/LejvW.png)

Alas I was not able to find other papers looking into this aspect, however it does look to me that the grit paper does point in the same direction; it found that grit was a much better predictor than ACT (itself considered a proxy for IQ) for CS1 success.

The humps paper mentiond in the question is not taken very seriously (most of these papers dont' cite it), and [even by its own authors themselves these days](http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-two-humps-programming-aptitude-test-canned-for-overzealous-conclusion/). It also does not seem to have been the first to observe the grade distribution phenomenon. I'm not entirely sure why it became popular on-line. Its strong language and/or the SE-effect maybe. For instance [this 2005 paper](https://doi.org/10.1145/1083431.1083474) noted the 30% to 40% failure rate as well.

---

Also here's a graph with the failure rate by country from the 2014 meta-analysis:

[![enter image description here](https://i.sstatic.net/xDyKL.png)](https://i.sstatic.net/xDyKL.png)

The authors note that the top 3 failure leaders Portugal, Germany and Brazil were represented by a pretty small sample in this study, so the data for those countries cannot really be relied upon. On the other hand, "of the 4 countries
which made 80% of the sample, Finland was found to have the lowest pass rate 57.7%". The other three countries in this 80% bulk of the data, US, UK, and Australia had statistically indistinguishable rates.

Also, the methodology should have been more upfront in this meta-analysis but they do say "Currently we have found in this study 3/10 students do not complete, or fail CS1." So they definitely count withdrawals as failure, basically, for the statics they reported.


---

### Answer by username_3 on 2017-12-15. Score: 2

According to the research of Elizabeth Patitsas, [the distribution is not bimodal](https://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html). She sampled 778 University of British Columbia CS courses, as well as 5 University of Toronto CS courses and found the vast majority to not be bimodal:

> An estimated 85.1% of the final grades in UBC's undergrad CS classes
> are normally-distributed. 5.8% of the classes tested as being bimodal,
> which isn't a whole lot more than the false positive rate I'd expect
> to see (5%).

You can read more about these results in the blog post linked above or in "[Evidence That Computer Science Grades Are Not Bimodal](https://dl.acm.org/citation.cfm?doid=2960310.2960312)" by Elizabeth Patitsas, Jesse Berlin, Michelle Craig and Steve Easterbrook.

To make things worse, when discussing this, [people who believe the distribution is bimodal are more likely to see a bimodal distribution in a normally distributed sampling](https://computinged.wordpress.com/2016/04/13/where-cs-teachers-see-bimodality-guest-blog-post-from-elizabeth-patitsas/):

> We delayed the debriefing until after our analysis was complete, on
> the assumption you’d want to know the preliminary results of the
> study. We indeed found that participants who agreed more strongly with
> the statement “Some students are innately predisposed to do better at
> CS than others” were statistically significantly more likely to label
> ambiguous distributions as bimodal.

> **Comments:**
> **username_1**: Nice find; the psychology of the observer explained (in the paper abstract):"A random half of participants were primed to think about the fact that CS grades are commonly thought to be bimodal; these participants were more likely to label ambiguous dis- tributions as \bimodal". Participants were also more likely to label distributions as bimodal if they believed that some students are innately predisposed to do better at CS. These results suggest that bimodal grades are instructional folklore in CS, caused by conrmation bias and instructor beliefs about their students."  
> **username_1**: I do have to wonder if the UBC data is representative though. They do mention a paper with an over-lapping set of authors (Zingaro et al.) in which the exam is turned into many fine-grained questions. So, if this practice is commonly applied at UBC (but not in other places), then UBC data is not representative. Ah, never mind the UBC data is from 1996 to 2013, while the Zingaro paper is from 2012, so not much chance it was applied unless it was "prior art" that nobody thought to publish about for years (unlikely).  
> **username_3**: @Fizz fair criticism which she acknowledges in her blog post. I guess we could do the same tests on username_2's data?  
> **username_1**: Good point on trying that with different data. The methodological question is what do we do with the large number of withdrawals. The Patitsas paper says NOTHING about that... which is probably the biggest question mark, now that I realized it.  
> **username_1**: I'm pretty sure the 2014 meta-analysis counted withdrawals as failed. That's because they have ~30% average failure for US (it has a graph by country as well), so with username_2's data, you'd definitely have to add F+W to get anywhere near that 30% number. Also, at some US institution I'm familiar with, the D's would be failure as well because the students need at least a C to enroll in subsequent CS classes. The Canadian failure rate in the meta-analysis was only 20%, by the way.  
> **username_1**: Yeah, the methodology should have been more upfront in the meta-analysis but they do say " Currently we have found in this study 3/10 students do not complete, or fail CS1." So they definitely count withdrawals as failure, basically.  

---

