<p>If you work for an industrial lab, it can be much, much easier to get a paper approved for release than to get code approved for release (even if the paper contains all the information needed to rewrite the code). Blame bureaucracy.</p>

CC BY-SA 2.5
2011-03-09T06:22:55.527Z
while this is of course true, I wonder if an appropriate change in culture driven by academic researchers would enable industrial researchers to pressure management to allow code release. After all, I suspect even getting a policy for  paper release must have taken some doing
Suresh Venkat
CC BY-SA 2.5
2011-03-09T12:11:18.833Z
At least at MSR, it's very easy to release research code. It's actually harder to use outside code, since MS is super-careful about respecting software licenses. Since many paper-related code drops come without an explicit license file, this can be a little annoying. Usually it just takes an email to the paper author to clear things up, but you can make it easier for industrial researchers by remembering to stick an OSI-approved license on your public code release.
Neel Krishnaswami
CC BY-SA 2.5
2011-03-09T13:07:55.023Z
@Neel: Do you mean "OSI-approved, but *not* GPL"?
Radu GRIGore
CC BY-SA 2.5
2011-03-09T16:01:22.043Z
The GPL is usually ok -- lots of us use Emacs! :) We're not allowed to use Affero GPL software, since its reciprocity conditions extend to anyone who interacts with the software (ie, it closes the web service loophole), and MS doesn't want to risk the possibility that some internal server running AGPL code might accidentally be made public-facing. But even the AGPL is better than no license at all, because picking a license actually makes the terms of sharing unambiguous.
Neel Krishnaswami
CC BY-SA 2.5
2011-03-08T21:53:23.893Z
2011-03-08T21:53:23.893Z
Peter Shor 
<p>Migrated and expanded from a comment:</p>

<p>I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from, e.g., ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution.</p>

<p>Of the remaining papers, a fair amount never had code to begin with. Of those, there's probably two main reasons. First are the papers whose main content is proof trees, typing rules with associated soundness proofs and the like. Of those, advances in mechanized metatheory have encouraged at least some authors to provide code in their theorem prover of choice (see Weirich's slides on POPLmark: <a href="http://www.seas.upenn.edu/~sweirich/talks/cambridge-09.pdf">http://www.seas.upenn.edu/~sweirich/talks/cambridge-09.pdf</a>). Second are those which are descended from the Bird-Merteens stuff (banannas &amp; co.). These are generally translatable into a functional language without too much work. However, I suspect that there's both typically a loss of generality, and that dealng with concrete issues of syntax and typing needlessly complicates things and makes it harder to follow the equational reasoning.</p>

<p>I wanted to substantiate my observations a bit, so did a rough count of the first two days of ICFP 2010. Of standard papers (i.e. not experience reports or invited talks), 12 out of 21 provided code of some sort. Three provided Coq (a fourth claimed a partial proof but did not publish it). Three proided Haskell. Three provided Agda. One provided Scheme, one provided Caml, and one provided Twelf. (Note that some provided code for more than one proof assistant, or for both a formalization and an implementation). Of the remaining papers, a few did work at a high enough level of abstraction that implementing it in a proof assistant would be a new paper in itself, and a fair number more did work that I suspect could have been implemented in a proof assistant using standard techniques, but which certainly would have taken a fair amount of work to do so. A few further papers claimed implementations/releases as future work.</p>

CC BY-SA 2.5
2011-03-09T13:00:52.570Z
2011-03-09T16:09:44.343Z
2011-03-09T16:09:44.343Z
sclv
<p>You believe that code should be published, but you ask why papers do not include code. These are two different things.</p>

<p>Most of the time, there is simply not enough room to publish a significant amount of code. In my research field (image processing) pseudocode or architecture information is often far more valuable and I have never found myself stuck due to the lack of code in a paper. It's often left as an exercise to the reader who grasped the article.</p>

<p>Yet there is a lot of code available to illustrate papers. Authors usually have a webpage and even if the reviewer doesn't get a chance to try and check the code itself, natural selection appears to work pretty well and authors who do not publish code are a lot less cited.</p>

CC BY-SA 2.5
2011-03-09T12:56:42.800Z
2011-03-09T12:56:42.800Z
sam hocevar
<p>This might have been asked sometime ago already, however I have always felt strongly about this so I will give my two cents. I have worked for years (not anymore) within the SAT community. Most researchers rarely publish their code. The paper is published along with the algorithm but it is very rare to see the actual code of the SAT solver (MAXSAT solver) etc, published along with the paper. </p>

<p>And the reality is that with the just the code published in the paper you will <strong>never</strong> have the chance to reproduce the author's experiments. Not only because the published code is not complete (of course) but also because even the published pseudo-code rarely translates semi-directly into what is actually implemented.</p>

<p>The reason behind this is hard to know and it might depend from researcher to researcher but mostly it is two fold.</p>

<ul>
<li><p>First, the researcher tends to work continually in a single solver publishing papers after papers on the same solver and incrementally adding new features which translate to new versions of the solver. There is an unhealthy obsession that competition will use <em>your</em> solver to further their careers by extending it and publishing papers without giving you due credit (meaning, co-authorship).</p></li>
<li><p>Secondly, some code really is (as with all software) written in a rush. Half-baked scripts. Untested features, etc. By publishing this code the researcher would feel he would be embarrassing themselves and damaging their reputation. </p></li>
</ul>

<p>I leave you with a recent reference on this from ACM:
<a href="http://cacm.acm.org/magazines/2011/5/107698-the-importance-of-reviewing-the-code/fulltext">http://cacm.acm.org/magazines/2011/5/107698-the-importance-of-reviewing-the-code/fulltext</a></p>

CC BY-SA 3.0
2011-07-08T07:47:08.793Z
2011-07-08T07:47:08.793Z
Paulo Matos
<p>Historically, scientific articles had to be printed on paper, and journals were shipped internationally. Each extra page used to add a significant cost, thus articles were subject to length limitations, and even simple working code usually takes way much space than an informal description.</p>

<p>Today there is no good reason not to include code in any kind of article that references an algorithm.</p>

<p>It may also useful to abandon print-oriented formats like pdf and postscript in favor of more semantically-aware formats (HTML with MathML or perhaps an opensource variation of Mathematica).</p>

CC BY-SA 2.5
2011-03-09T16:56:01.127Z
+1 for first two paragraphs, -1 for last paragraph.  You can take away my LaTeX when you pry it from my cold dead hands.
Jeffε
CC BY-SA 2.5
2011-03-09T17:05:58.967Z
There are now plenty of tools available to enable literate LaTeX programming...
sclv
CC BY-SA 2.5
2011-03-09T16:16:55.810Z
2011-03-09T16:16:55.810Z
Antonio Valerio Miceli-Barone
<p>Here is a well argued article by David Donoho and Jonathan Buckheit that I read in grad school which touches on exactly this topic from the point of view of wavelet researchers:</p>

<p><a href="http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pdf">"WaveLab and Reproducible Research"</a></p>

<p>Their idea was even more ambitious, to provide code for reproducing all the figures in their papers in a convenient Matlab package.</p>

<p>I really like their idea but I think the issues are obvious.</p>

<p>(1) It is extra work (cleaning up the code, making at least a rudimentary user interface, writing some documentation, providing some support when people inevitably run into problems)</p>

<p>(2) It is not really required/expected by most conferences/reviewers</p>

<p>But I can't help but feel that the CS research community would benefit if there was an expectation of making the code and data used in any publication publicly available in a usable format. I admit that I haven't done so myself even when the amount of work involved would have been manageable. I think it is just hard to make yourself put in the extra effort when there is no external push.</p>

CC BY-SA 3.0
2011-04-17T02:46:05.763Z
code does not need to be good, clean, or reusable. but it does need to be reproducible in order to be considered 'science'. There was a good article in nature  by Nick Barnes (2010) [Publish your computer code: it is good enough](http://www.nature.com/news/2010/101013/full/467753a.html) arguing this point.
David LeBauer
CC BY-SA 2.5
2011-03-09T13:51:30.567Z
I think an expectation of either (a) pseudocode or (b) code available, at least on websites, would be very useful for the CS research community. Not all research code has high useability, and I'm not sure whether making buggy code available would be a good thing or not.
Peter Shor 
CC BY-SA 2.5
2011-03-09T20:30:10.650Z
I've talked to paper authors who haven't released their code for similar reasons -- in the state it's in, they'd feel guilty if anybody actually tried to do anything with it.
sclv
CC BY-SA 2.5
2011-03-09T04:58:18.950Z
2011-03-09T04:58:18.950Z
Mugizi Rwebangira
<p>In my academic career, I've read quite a few academic papers on various computer science topics.  Many of which involve an implementation and some assessment of that implementation, yet I have found that very few of them actually publish the code they used.</p>

<p>To me, the benefits of including the actual implementation would be significant, namely:</p>

<ul>
<li>Extension of trust or reproducability (just test it yourself!)</li>
<li>Clarification of ambiguities (particularly for papers written by non-native speakers)</li>
<li>Reuse of code for applications</li>
</ul>

<p>So why is it that so few papers actually include any code?</p>

<p>I suppose that it might be the intention of the organization behind the paper to utilize the implementation in their own applications, and thus would not want to release it, but if that's the case, why even write the paper?</p>

CC BY-SA 2.5
2011-03-08T22:11:41.710Z
*So why is it that so few papers actually include any code?* Because the reviewers tolerate it for some reason, even if there is no good reason to hide the implementation. We are the reviewers, we can change it.
Jukka Suomela
CC BY-SA 2.5
2011-03-08T22:24:10.597Z
I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution. Of the remaining papers, a fair amount never had code to begin with.
sclv
CC BY-SA 2.5
2011-03-09T03:32:43.177Z
@sclv, your comment can be an answer! :)
Kaveh
CC BY-SA 2.5
2011-03-09T12:06:13.817Z
Research results should be open (and free), and so should code. By some definitions, science only happens when hypotheses are falsifyable and experiments reproducible, so you could argue that publications basing on code that is not published are not scientific work.
Raphael
CC BY-SA 2.5
2011-03-09T14:23:08.780Z
@Raphael Releasing the same code would only help establish repeatability, not reproducibility. Running the exact same code in the exact same experiment is hardly independent. Science requires reproducibility, not repeatability.
Mark Reitblatt
CC BY-SA 2.5
2011-03-09T19:56:46.703Z
I was thinking of results that talk about properties of the code rather than its results (in which case you are right, of course). Say, the authors conduct experiments on some implementation of the presented algorithm and compare runtimes by examining graphs (they call it "Algorithm Engineering"). Here, code provides reproducibility.
Raphael
CC BY-SA 2.5
2011-03-10T02:04:30.897Z
@Raphael The same distinction holds here. Running the same code to get the same results (performance figures) is repeatability. Independently implementing the algorithm and getting commensurate results would be reproducibility.
Mark Reitblatt
CC BY-SA 2.5
2011-03-10T17:52:35.650Z
In your scenario, right. But read again: I wrote "properties of the code", not "properties of the algorithm".
Raphael
CC BY-SA 2.5
2011-03-10T20:13:09.383Z
@Raphael Papers directly about code sometimes blur the line. But if we're claiming to make a scientific rather than engineering contribution, then the code itself corresponds to an experiment, not a scientific object of study. Engineering has different requirements.
Mark Reitblatt
<soft-question><research-practice><paper-review>
Code in Academic Papers
Kevin Dolan
# Code in Academic Papers

(Asked by: username_0 on 2011-03-08)

In my academic career, I've read quite a few academic papers on various computer science topics. Many of which involve an implementation and some assessment of that implementation, yet I have found that very few of them actually publish the code they used.

To me, the benefits of including the actual implementation would be significant, namely:

* Extension of trust or reproducability (just test it yourself!)
* Clarification of ambiguities (particularly for papers written by non-native speakers)
* Reuse of code for applications

So why is it that so few papers actually include any code?

I suppose that it might be the intention of the organization behind the paper to utilize the implementation in their own applications, and thus would not want to release it, but if that's the case, why even write the paper?

**Tags:** soft-question, research-practice, paper-review
> **Comments:**
> **username_1**: *So why is it that so few papers actually include any code?* Because the reviewers tolerate it for some reason, even if there is no good reason to hide the implementation. We are the reviewers, we can change it.  
> **username_2**: I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution. Of the remaining papers, a fair amount never had code to begin with.  
> **username_3**: @sclv, your comment can be an answer! :)  
> **username_4**: Research results should be open (and free), and so should code. By some definitions, science only happens when hypotheses are falsifyable and experiments reproducible, so you could argue that publications basing on code that is not published are not scientific work.  
> **username_5**: @Raphael Releasing the same code would only help establish repeatability, not reproducibility. Running the exact same code in the exact same experiment is hardly independent. Science requires reproducibility, not repeatability.  
> **username_4**: I was thinking of results that talk about properties of the code rather than its results (in which case you are right, of course). Say, the authors conduct experiments on some implementation of the presented algorithm and compare runtimes by examining graphs (they call it "Algorithm Engineering"). Here, code provides reproducibility.  
> **username_5**: @Raphael The same distinction holds here. Running the same code to get the same results (performance figures) is repeatability. Independently implementing the algorithm and getting commensurate results would be reproducibility.  
> **username_4**: In your scenario, right. But read again: I wrote "properties of the code", not "properties of the algorithm".  
> **username_5**: @Raphael Papers directly about code sometimes blur the line. But if we're claiming to make a scientific rather than engineering contribution, then the code itself corresponds to an experiment, not a scientific object of study. Engineering has different requirements.  

---

## 6 answers

### Answer by username_14 on 2011-03-09. Score: 17 | Accepted answer

Here is a well argued article by David Donoho and Jonathan Buckheit that I read in grad school which touches on exactly this topic from the point of view of wavelet researchers:

["WaveLab and Reproducible Research"](http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pdf)

Their idea was even more ambitious, to provide code for reproducing all the figures in their papers in a convenient Matlab package.

I really like their idea but I think the issues are obvious.

(1) It is extra work (cleaning up the code, making at least a rudimentary user interface, writing some documentation, providing some support when people inevitably run into problems)

(2) It is not really required/expected by most conferences/reviewers

But I can't help but feel that the CS research community would benefit if there was an expectation of making the code and data used in any publication publicly available in a usable format. I admit that I haven't done so myself even when the amount of work involved would have been manageable. I think it is just hard to make yourself put in the extra effort when there is no external push.

> **Comments:**
> **username_6**: I think an expectation of either (a) pseudocode or (b) code available, at least on websites, would be very useful for the CS research community. Not all research code has high useability, and I'm not sure whether making buggy code available would be a good thing or not.  
> **username_2**: I've talked to paper authors who haven't released their code for similar reasons -- in the state it's in, they'd feel guilty if anybody actually tried to do anything with it.  
> **username_15**: code does not need to be good, clean, or reusable. but it does need to be reproducible in order to be considered 'science'. There was a good article in nature  by Nick Barnes (2010) [Publish your computer code: it is good enough](http://www.nature.com/news/2010/101013/full/467753a.html) arguing this point.  

---

### Answer by username_6 on 2011-03-08. Score: 16

If you work for an industrial lab, it can be much, much easier to get a paper approved for release than to get code approved for release (even if the paper contains all the information needed to rewrite the code). Blame bureaucracy.

> **Comments:**
> **username_7**: while this is of course true, I wonder if an appropriate change in culture driven by academic researchers would enable industrial researchers to pressure management to allow code release. After all, I suspect even getting a policy for  paper release must have taken some doing  
> **username_8**: At least at MSR, it's very easy to release research code. It's actually harder to use outside code, since MS is super-careful about respecting software licenses. Since many paper-related code drops come without an explicit license file, this can be a little annoying. Usually it just takes an email to the paper author to clear things up, but you can make it easier for industrial researchers by remembering to stick an OSI-approved license on your public code release.  
> **username_9**: @Neel: Do you mean "OSI-approved, but *not* GPL"?  
> **username_8**: The GPL is usually ok -- lots of us use Emacs! :) We're not allowed to use Affero GPL software, since its reciprocity conditions extend to anyone who interacts with the software (ie, it closes the web service loophole), and MS doesn't want to risk the possibility that some internal server running AGPL code might accidentally be made public-facing. But even the AGPL is better than no license at all, because picking a license actually makes the terms of sharing unambiguous.  

---

### Answer by username_2 on 2011-03-09. Score: 13

Migrated and expanded from a comment:

I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from, e.g., ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution.

Of the remaining papers, a fair amount never had code to begin with. Of those, there's probably two main reasons. First are the papers whose main content is proof trees, typing rules with associated soundness proofs and the like. Of those, advances in mechanized metatheory have encouraged at least some authors to provide code in their theorem prover of choice (see Weirich's slides on POPLmark: <http://www.seas.upenn.edu/~sweirich/talks/cambridge-09.pdf>). Second are those which are descended from the Bird-Merteens stuff (banannas & co.). These are generally translatable into a functional language without too much work. However, I suspect that there's both typically a loss of generality, and that dealng with concrete issues of syntax and typing needlessly complicates things and makes it harder to follow the equational reasoning.

I wanted to substantiate my observations a bit, so did a rough count of the first two days of ICFP 2010. Of standard papers (i.e. not experience reports or invited talks), 12 out of 21 provided code of some sort. Three provided Coq (a fourth claimed a partial proof but did not publish it). Three proided Haskell. Three provided Agda. One provided Scheme, one provided Caml, and one provided Twelf. (Note that some provided code for more than one proof assistant, or for both a formalization and an implementation). Of the remaining papers, a few did work at a high enough level of abstraction that implementing it in a proof assistant would be a new paper in itself, and a fair number more did work that I suspect could have been implemented in a proof assistant using standard techniques, but which certainly would have taken a fair amount of work to do so. A few further papers claimed implementations/releases as future work.


---

### Answer by username_10 on 2011-03-09. Score: 12

You believe that code should be published, but you ask why papers do not include code. These are two different things.

Most of the time, there is simply not enough room to publish a significant amount of code. In my research field (image processing) pseudocode or architecture information is often far more valuable and I have never found myself stuck due to the lack of code in a paper. It's often left as an exercise to the reader who grasped the article.

Yet there is a lot of code available to illustrate papers. Authors usually have a webpage and even if the reviewer doesn't get a chance to try and check the code itself, natural selection appears to work pretty well and authors who do not publish code are a lot less cited.


---

### Answer by username_11 on 2011-07-08. Score: 8

This might have been asked sometime ago already, however I have always felt strongly about this so I will give my two cents. I have worked for years (not anymore) within the SAT community. Most researchers rarely publish their code. The paper is published along with the algorithm but it is very rare to see the actual code of the SAT solver (MAXSAT solver) etc, published along with the paper.

And the reality is that with the just the code published in the paper you will **never** have the chance to reproduce the author's experiments. Not only because the published code is not complete (of course) but also because even the published pseudo-code rarely translates semi-directly into what is actually implemented.

The reason behind this is hard to know and it might depend from researcher to researcher but mostly it is two fold.

* First, the researcher tends to work continually in a single solver publishing papers after papers on the same solver and incrementally adding new features which translate to new versions of the solver. There is an unhealthy obsession that competition will use *your* solver to further their careers by extending it and publishing papers without giving you due credit (meaning, co-authorship).
* Secondly, some code really is (as with all software) written in a rush. Half-baked scripts. Untested features, etc. By publishing this code the researcher would feel he would be embarrassing themselves and damaging their reputation.

I leave you with a recent reference on this from ACM:
<http://cacm.acm.org/magazines/2011/5/107698-the-importance-of-reviewing-the-code/fulltext>


---

### Answer by username_12 on 2011-03-09. Score: 7

Historically, scientific articles had to be printed on paper, and journals were shipped internationally. Each extra page used to add a significant cost, thus articles were subject to length limitations, and even simple working code usually takes way much space than an informal description.

Today there is no good reason not to include code in any kind of article that references an algorithm.

It may also useful to abandon print-oriented formats like pdf and postscript in favor of more semantically-aware formats (HTML with MathML or perhaps an opensource variation of Mathematica).

> **Comments:**
> **username_13**: +1 for first two paragraphs, -1 for last paragraph.  You can take away my LaTeX when you pry it from my cold dead hands.  
> **username_2**: There are now plenty of tools available to enable literate LaTeX programming...  

---

