In recent years, scientists may have inadvertently given up on a key component of the scientific method: reproducibility. That's an argument that's being advanced by a number of people who have been tracking our increasing reliance on computational methods in all areas of science. An apparently simple computerized analysis may now involve a complex pipeline of software tools; reproducing it will require version control for both software and data, along with careful documentation of the precise parameters used at every step. Some researchers are now getting concerned that their peers simply aren't up to the challenge, and we need to start providing the legal and software tools to make it easier for them.
In the past, reproduction was generally a straightforward affair. Given a list of reagents, and an outline of the procedure used to generate some results, other labs should be able to see the same things. If a result couldn't be reproduced, then it could be a sign that the original result was so sensitive to the initial conditions that it probably wasn't generally relevant; more seriously, it could be viewed as a sign of serious error or fraud. In any case, the ability to reproduce a given result is key to its general acceptance and, since a successful experiment is often the foundation of further research, often essential for pushing a field forward.
But, when it comes to computational analysis, both the equivalent of reagents and procedures have a series of issues that act against reproducibility. The raw material of computational analysis can be a complex mix of public information and internally generated data—for example, it's not uncommon to see a paper that combines information from the public genome repositories with a gene expression analysis performed by an individual research team.
A lot of this data is in a constant state of flux; new genomes are being completed at a staggering pace, meaning that an analysis performed six months later may produce substantially different results unless careful versioning is used. In some cases, the data resides in proprietary databases. Some of this work may run up against the issues of data preservation, as older information may reside on media that's no longer supported or in file formats that are difficult to read.
And that's just the data. An analysis pipeline may involve dozens of specialized software tools chained together in series, each with a number of parameters that need to be documented for their output to be reproduced. Like the data, some of these tools are proprietary, and many of them undergo frequent revisions that add new features, change algorithms, and so on. Some of them may be developed in-house, where commenting and version control often take a back seat to simply getting software that works. Finally, even the best commercial software has bugs.
The net result is that, even in cases where all the data and tools are public, it may simply be impossible to produce the exact same results. And that has serious implications for the use of reproducibility in every sense: validating existing results by repeating them, determining their robustness by performing a distinct but overlapping analysis, and using the results as a basis for further research. Perhaps most significantly, without a way of determining precisely what was done, it's impossible to identify the differences that sometimes cause similar analyses to produce different results.
At this point, one might be tempted to make a misguided argument that the advent of computational tools means that it's time for a major revision of the scientific method. But reproducibility became a key component of that method precisely because it's so useful, which is why a number of people are pushing for the field to adopt standards to ensure that computational tools conform to the existing scientific method.
Victoria Stodden is currently at Yale Law School, and she gave a short talk at the recent Science Online meeting in which she discussed the legal aspects of ensuring that the code behind computational tools is accessible enough for reproducibility. The obvious answer is some sort of Creative Commons or open source license, and Stodden is exploring the legal possibilities in that regard. But she makes a forceful argument that some form of code sharing will be essential.
Still, this sort of disclosure, as with other open source software, should provide a key benefit: more interested parties able to evaluate and improve the code. "Not only will we clearly publish better science, but redesigned and updated code bases will be valuable scientific contributions," Stodden said. "Over time, we won't stagnate forever on one set of published code."
Even if we solve the legal and computational portions of the problem, however, we're going to run into issues with the fact that many of the people who use computational tools understand what they do, but don't feel compelled to learn the math behind them. That's where a paper in the latest edition of Science comes in. Its author, Jill Mesirov of the Broad Institute, describes how many biologists aren't well versed in computational analysis, but are increasingly reliant on tools created by those who are; she then goes on to describe one type of solution, called GenePattern, that she and her colleagues put together with the help of Microsoft Research.
The idea is that the researchers that rely on computational techniques as part of their day-to-day activities need an entire "reproducible research system" that will make it easier for them to document the sources of their data and the analyses performed on it. The system they've designed shares features with rapid application development environments, as it graphically represents modular computational tools, which can be ordered to create an analysis pipeline, and the individual settings for each can be tweaked. Once complete, the user can trigger the analysis to run; the system documents all of the relevant settings and software information.
Microsoft got involved because Mesirov was apparently inspired by the fact that it's possible to embed a spreadsheet in a word processing document and leave it accessible to all the standard tools for manipulating that data. So, the GenePattern software includes a Word plugin that can embed the output of any data created using these tools, such as a table or chart. When clicked, the full GenePattern environment makes all the information about the experiment—the software and settings used, for example—accessible to the reader. It's even possible to rerun the same analysis, a sort of point-and-click reproducibility.
There are clearly some limitations to this implementation. Although the software is cross-platform, the plug-in is not. Publishers have standardized on PDFs (not Word docs) for distributing research papers, and GenePattern is limited to those analysis tools that have been adapted to work with it. But there's no question that it's the right sort of approach, as it implicitly recognizes that just about anyone doing science is now doing some form of computation, and very few of them are likely to be fully cognizant of everything that's involved in making sure their documentation is sufficient to ensure reproducibility. Unless the tools that do the analysis are designed to do so for them, a lot of this critical information is likely to be lost to the rest of the scientific community.
If a result is that sensitive to a particular implementation of an algorithm, the result is not science. You should be able to publish your code and data, and somebody 20 years from now should be able to re implement the process in whatever tools are available then and get similar results. So what if their new tool implements a statistical algorithm slightly differently - the results should be similar. If not, it was not a robust result to begin with.
The code/comments/paper should make abundantly clear the reasons and expected results of every processing step of whatever programs you use - sufficient to allow re-implementation on any other platform with similar processing capabilities.
This sounds like someone saying, "I need you to mail me the original beaker you used to do this, it might have some different dirt on it than mine have."
If someone else can't reproduce similar results from the same input, you're trusting your tools too much.
And if someone can't reproduce it with similar data, you're trusting your data too much.
I think this is a particularly big problem in mathematics.
Nowadays, many proofs are being done with the help of computer algebra software to solve systems of equations that would simply be impossible for humans to solve by hand (time/accuracy), but it is very difficult to break into a new field if you don't have code and scripts handed down from one researcher to another. It's like if you had to compile "Linux From Scratch" to start each project, but there were no Makefiles or autotools and gcc barely worked.
Thats about what I wanted to say. An algorithm's results should not be dependent on a particular implementation. If two implementations of the same algorithm give you different results, one or maybe both of them are broken. The only way that there should be confusion if it is unknown precisely what algorithm was used (and it sounds like this might be the case), which is just bad science.
Philippe23, it's like saying "You used a beaker for this experiment, now I need a beaker to reproduce it" If beakers have become obsolete and no-one is around with the know-how to recreate them, then it's nearly impossible to reproduce the experiment without the necessary tools.
When a beaker can be stored in the space of 1"x0.5" flash drive, it seems kind of ridiculous that a beaker cannot be found anywhere.
Agreed, but Stodden pointed out that this is also an argument for sharing code and data. Without the sharing aspect, you can't tell either how similar the data and approach are, or where the problems entered the analysis. All you get is confusion.
While I like this story and agree with the overall point it makes, I think the topic is somewhat sensationalistic and "crying wolf". An equation can be solved and the solution can be reproduced on other machines with different compilers and the difference between the two solutions is typically one or two orders of machine error which is meaningless in a 32 bit calculation. Simulations on the other hand, are solving these equations in multi-dimensional space which the math often fails to address- what may be well-posed in a single equation or a set of coupled equations, becomes unknown in multi-dimensions. This leads to the time tested review of "does the result make sense". I consider simulations to be numerical experiments analogous to laboratory experiments and exact reproducibility does not exist in the same sense as reproducing a well-posed math solution. Because of the unknown of the well-posedness and the grid resolution, skewness, and spatial and temporal solution order, there will always be tweaks and constraints and these are too numerous to document in a journal article.
If the code is placed in a open repository where people can sift through it with a fine tooth comb they can go about tweaking all same things the authors did that were necessary to get to the final result. This is a huge waste of time unless there are large discrepancies which are truly attributed to new physics being revealed by the simulation. The reason I consider this to be a waste of time is that there are numerous other simulation codes that are meant to capture the same physics, so it is rather trivial to reproduce the results on a completely different platform. Trivial in the sense it is much cheaper than, for example, building another LHC just to reproduce results.
Any bold new physics discoveries made with a simulation are going to be simulated again and again on different systems, and if the results are far different, then the scientific system works by refuting the original claims.
A couple of underlying problems with the current publication system are: 1) you don't get published for verifying a result. This is a huge problem both numerically and experimentally as I feel that doing the experiment in a separate laboratory and getting a similar results is just as important as the discovery but reviewers don't see it this way. 2) you don't get published for conducting benchmarks. I think NSF should devote a fair chunk of their computational grant money to benchmark development of multi-physics, multi-dimensional codes. The ones code writers use now are good enough to get us out of the gate but what is really lacking, and is needed to show the difference all the tweaks make on those really large codes, is the large-scale, complicated bench mark.
It's also worth noting that the compiler used and the hardware it is run on can have significant numerical implications. Think pentium bug, and compare something like intel fortran compiler to the gfortran compiler.
Compiler options are yet another can of worms when dealing with this issue, as there are a whole class of optimizations that can be "unsafe." Not to mention those that toss IEEE floating point compliance out the window. And hardware vs. software computations can be an issue as well, especially for floating point numbers.
Identical code, compiled with identical options with the identical compiler (itself compiled with the same options), on identical machines, right down to the chipset, should produce identical results with identical input. Change any one of these slightly, and especially in floating point computations, errors can stack up quite rapidly.
As an example, until GCC 4.4, using doubles to calculate something as simple as 100! resulted in an error after about the tenth digit (and there are a lot of digits in 100!). It was rather ridiculous how quickly the errors added up. Same code, same machine, using the 10.0.015 version of the intel compiler, and the value matched the correct value to all digits. The latest GCC gives the correct value.
So, basically what I'm trying to say is, the compiler can play a huge role in computational methods, and unfortunately, it is very poorly understood by most scientists. And since numerical computations are the bread and butter of scientific codes, this role can be significantly magnified, especially if a long chain of computations is executed on a floating point value that has already started to skew.
True, it can be stored, but when you want to use it couple of years later it turns out that it requires a proprietary library X, made by company Y, only available on platform Z. We had this problem with old code written 10 years ago to run on SGI.
i too feel this is a huge issue. That is why i walked away from Matlab and the like and started implementing my algorithms in Python using SciPy/NumPy and other libraries. Also, i plan to release all of my code under GPL and keep it in a public repository in the exact versions used in my simulations.
Cool coverage. I didn't see this one coming! But in my experience (i work with scientists but I'm not one... yet), it's true that a lot of computing is of the lazy/rushed, "hey grad student, just f-ing make it work NOW" variety.
4. institutionalized rules for source code licensing.
Hopefully some good rules can be developed for making deadlines on when this information has to be published, to balance the need for openness with the "right" of a research team to not get "beaten to the punch" by someone else using their data.
vibedog is right about the "no honor in reproducing" aspect.
Also, in the bio-sciences, a HUGE problem is that the NIH generally refuses to pay for computers, so labs (that are also not CS or computer admin experts) cobble together old junk to make barely-workable ad-hoc storage and processing capacity with the simplest or most immediate software they can find.
If a result is that sensitive to a particular implementation of an algorithm, the result is not science.
This article makes it seem as if science is done in magical mythical towers of black boxes of which we'll never be able to see inside. The truth of the matter is: the raw data should dictate the result, not the analysis of the data. The same statistical conclusion should be drawn from the raw data, as long as all parameter inputs are reasonable. It should not matter which version of software you use. Sure, the numbers won't be exactly the same. But the same general process should be done.
On the flip side, one should always be aware of what parameters do what on your analysis. One should never be 'plugging numbers in,' without an understanding of what one is doing. Unfortunately, for most science, those doing the work [grad students, research assistants] don't know what the parameters are doing, and then one gets the scientific meltdown described here.
Another issue that was briefly touched on was licensing. Our University won't allow anything to be published under the GPL, period. I tried a few years ago and got roundly smacked down. They wanted to keep the option of commercialization, but still let me publish. In the end, I was able to use the OSL (Open Software License), which is very similar.
A copy-left license like this is the most compatible license with the way that science is done in academia. Another alternative would be the original MIT license (the one with the attribution requirement). If code is going to be published (in a journal), there needs to be an attribution requirement, just like any other citation.
Another problem that I see everyday is in a field like bioinformatics. Right now it's a lot of manual work. Pulling data from database A, putting that in to Excel, calculating a number, comparing that with database B, etc... Most people in this field don't treat it like scientific process at all. There is no equivalent of the lab notebook for bioinformatics. If you had to go back and reconstruct what happened, it's impossible. I come from a web-lab background, so I find myself writing little one off bash scripts to do almost everything so that I can always look back at what I did to get my results.
Since a failed attempt at reproduction, or the generation of inconsistent results implies that something isn't right, but doesn't tell us what it is, scientists need to be able to look for errors, hidden dependencies, or what have you everywhere in the experimental system. If the software version, hardware, and parameters used in the original result are long gone, then checking up on that result becomes much, much more difficult.
At least, that's how I read the story, and particularly in light of Dr. Jay's comments above.
Exactly. My office-mates know when I'm working on computations, because I start cursing, muttering under my breath, and banging the desk with my fist.
This is not just the normal frustration of programming, it is having a set of (usually very expensive) software tools that also have piss-poor documentation and no transparency.
I've gone back and added another sentence to really make this explicit, since it's clear from the comments here that it wasn't obvious enough as i originally wrote it. It's actually touched on in several places, but it's always tough to translate a story with a few interwoven concepts into a linear medium without scattering at least one of the concepts and diluting its impact accordingly. This is the one that got stuck being scattered in this case.
Which is somewhat ironic, since it's actually something i discussed with Stodden in detail, and she had a significant influence on my thoughts here.
First, I'll echo the comments of everyone else here, science shouldn't rely on the numerics you are using. If you are dealing with a chaotic system (e.g. the 10th digit in 100 matters) then you had better not be reliant on the exact result, you should be reliant on the statistics of the system, and those shouldn't change from on compiler/chip/OS to the next.
That said, I think the issue is as much about makeing science easier. I used to have problems when I revisited old work *I* had done. I had since modified some of the code I had written to work with a different set of files. As a result it was a nightmare to find all of the changes and revert back to them. I don't use version control... I know I should... but I don't. For now I'm making a copy of my entire script library (it's only a few MB) whenever I "finish" a project, so that when I go back to it I know I can re-run everything.
One of the problems with the approach listed (clicking on a table to pull up and re-run all of the analysis) is that to generate that 5 line table you might require 100GB of data... it hardly seems justifiable to transfer all 100GB of data around anytime you want to make a copy of that table, so you are stuck not being able to just click and re-run anyway. You certainly can't publish a 100GB paper in most academic journals, though it would be good to have an enormous data repository for this sort of archiving.
First off, the first thing that pops in my head when reading this article is the frustration around using something like Gaussian. It's a software package for modeling molecules, and depending on how you work it --- it does a very good job of it.
The logarithms, however, are locked down and unknown for the most part to the scientist user. It's very hard to comment on the code when you don't know exactly how it works. Even the method nomenclature is nonsensical, all we're left with are black boxes. A lot of computational chemists have their hands tied, and can't really do anything about it. Even trying to look at the computer code itself is fruitless, b/c it is written to be purposefully confusing. Mind you, these scientists spend their career studying quantum mechanics, so it's not like they're too dumb to get the core concepts. The company simply refuses to go public with the code. As such, scientists can not improve on it at all. It really does blow.
And the scientific method will say that is not repeatable. 2 + 2 is always four, and if your compiler/hardware says otherwise, then it is at fault, not the scientific method. There is no reason to change the scientific method, as the article states.
What sentence did you add? It complete escaped me and I read the article after you did the update. At first I thought you were arguing for preserving programs and hardware into antiquity, rather than arguing for transparency into what the programs and hardware did.
"Perhaps most significantly, without a way of determining precisely what was done, it's impossible to identify the differences that sometimes cause similar analyses to produce different results."
The issue in the case of some of the climate papers has been refusal to release the code or even the algorithm, or in some cases even the raw data. In fact, the CRU temperature series are still not reproducible, nor has anyone ever exactly reproduced the MBH98 results, and Mann has always refused to reveal the algorithm.
So its not just a technical issue, its a debate about what should count as publication. In the climate wars you have one side saying they want to see the raw data and the code applied to it, as it was used. Partly they do not trust the code to be correct and bug free, partly they may not even trust that this was the code and this was the data used.
The other side says a bunch of things. One is, go make your own code if you want to check our results, go get your own raw data. Another is, we are too busy doing science to make our code into turnkey packages.
The problem is, back in the days when you were applying well known algorithms from the public domain to your data, all you had to do was say which, and then supply the data and reproduction of results was possible. However, now you are saying (for instance) that I wrote my own Fortran (yes, they really did write the stuff in Fortran) and what it shows, when applied to my raw data, is the following principal components.
How are you to assess this claim if you cannot see the raw data and the code. Its a complicated one. Consider Lonnie Thompson's raw data. Allegedly those ice core readings show various things. But their raw values have never been published or documented. How much weight should we place on them? As the Polar Urals stuff showed when they were finally all recently released, things may look very different when examined by outsiders from the way they looked to the original author.
I'm not sure what you are getting at here. Fortran is the norm in many sciences. Just as scientists don't have time to make all of their work turn-key (and I think that is in some ways doing a disservice to themselves as well as everyone else), they don't have time to constantly be learning the latest and greatest computer language. Fortran still has some nice features for certain tasks (mind you I hate the stuff, but I use it occasionally, and I don't begrudge others using it).
I agree that data should be public, but there are cases where that may not be possible.
I think the current situation is far worse than most of you are assuming. We're not talking about differences on the level of compiler flags or linear algebra libraries. After you run the code and get numbers, you have to perform analysis to answer questions such as: "Are these two molecules bonded?" , "How much gas is adsorbed on this surface?" , or "Which of these crystal structures/protein conformations/etc are most likely to occur?" In order to answer these type of questions, you must often make a chain of assumptions in performing the analysis. Currently, this type of information may or may not be published, which can allow systematic blunders such as those at CRU. This lackadaisical attitude also hinders criticism of others' work. For example, people may accept the simple argument that an alternative configuration may be more stable, but there may not be an energetically favorable pathway from any known configuration to the supposed configuration.
Beyond reproducibility, there seems to be little concern for statistical significance in many computational papers. This is particularly unforgivable because running another simulation should be a one-click process. I think (the technical portion of) both of these problems could be solved by vastly increasing automation, which would practically necessitate a move away from languages such as C and Fortran.
Greg Wilson (University of Toronto and http://www.third-bit.com) has been writing and working on this issue for some years now. He's the creator of the Software Carpentry course (http://software-carpentry.org) for scientists, which I thoroughly recommend.
I'm curious why you don't make use of GAMESS?
For those who aren't aware of it, while not FOSS, it is a free program with nonclosed source(they retain distribution rights and in order to get it from their site they want an email address, but as far as I know it's so that the group can get usage numbers for funding). As far as I know it possesses all of the functionality of Gaussian, and is of comparable speed(although those who try and publish benchmarks are banned by Gaussian from using their code). It's pretty well documented too, with literature references for the methods, etc.
See previous posts on this subject. This is just the most basic of issues. The next comes from the use of statistics instead of scientific method to test drugs, global warming and all kinds of other things instead of putting in the effort to do it right and actually understand what's happening instead of side effects that they likely haven't controlled because they're using statistics to smooth the data instead of controlling variables.
The point is that in most of these cases the ultimate goal should still be to control all of the variables. Yes, it's very very hard, but we won't know for sure that our science is right until they do.
I think what you're describing can't even rightly be called statistics, but sadly, it's pretty common.
It's similar to Genepattern in a way in that it allows you to write your processing code alongside the rest of your publication/notes and have all of it directed into a unified LaTeX document.
Every time you rebuild the document, the charts, data, etc. can be rebuilt before being inserted.
(Obviously you run into problems for huge simulations and very large datasets in terms of processing time, but it's otherwise priceless for documentation).
There's a whole family of similarly aimed tools called noweb, from the literate programming movement.
I think the previous thread where you posted this showed pretty clearly that you don't know much about science or statistics. The way you have it, Boltzmann was a fraud, QM is magic, and don't even touch population genetics, it's a waste of time.
But Geminiman is sure that there's 'real' science out there somewhere, somewhen. We just haven't done any since at least the 20th century, perhaps earlier.
Then we'll never know for sure that our science is right, because there's no way to know that you've identified (never mind controlled) all the variables (nor even only the relevant ones) unless you already know everything. There's a reason Popper's work was so influential (even if ultimately misguided, IMO). I suggest you read it sometime.
I think the The history of SAGE ( http://sagemath.blogspot.com/2...are-and-me-very.html ) is relevant as an anecdote here. It went a long way in justifying to me the critical importance of open source in scientific computing. Not only does the source provide a means of auditing the means take to achieve results, it makes it possible for researchers to extend the system to incorporate their own work, and as part of the spirit of science, share not only the knowledge gained from their research, but the means to reproduce it and extend it.
At the moment you write your paper, you have all your code and data that you used. Zip it up, and submit it with the paper. If the journal you're submitting to doesn't make a bit of data space available, then *that's where the problem is*.
The Royal Society, for example, provides a fair bit of data space (which is made publicly available with the paper) with every submission. This is the only way of doing it.
Then, the only issues you'll encounter are those of potentially porting it to a different platform, but that is at least possible with some effort (I used to work in a bioengineering lab at UCSD doing exactly that for biological models of the heart), whereas if the data and code is not available, then it is utterly unreproducible. And if you wait until after publication to request the data, then it might already be too late (data gets lost all the time).
This article doesn't understand the scientific method, or science, or the natural world very well. Reproducibility isn't, as the first two commenters say, all about getting identical results. It's about getting good enough results. If you get identical results, you would have more than enough reason to be suspicious. And the point of bringing in computing to the analysis of results is to collate massive amounts of data to produce easily digested trends: you absolutely *should* get a variety of samples to analyze from which trends may be generated. These are the results: not the samples.
And similarly, if you have to use identical equipment in order to achieve reproducibility, then it would be appropriate to question the conditions of testing.
There is a whole range of topics dedicated to robustness in numerical computing. Any serious computation experiment will include safeguards, checks and other forms of verification to account for numerical instability in their algorithms. Hell, even some non-mission critical applications such as games include employ these techniques.
At the moment you write your paper, you have all your code and data that you used. Zip it up, and submit it with the paper.
That's not so easy. The "code" might include big libraries and software packages installed on the supercomputer by system administrators. It's not just a file I can burn to a disc. It's a whole package and I'm not the one who installed it or knows where all the important bits live in the computer system.
The "code" might also be proprietary. I have a license to use Mathematica. I don't have a license to publish copies of the Mathematica installation disc. I also have licenses to use niche software packages for my research. I can't publish them myself, and even if I tell you the name of the publisher they might be out of business by the time you read my paper.
The "code" can also require a lot of manual intervention. In between my raw data and my manuscript, I do a lot of manipulations to convert formats, perform regressions, and adjust plots. The save files have the final states of the data, but not scripts to automate all the steps that I did manually.
The solution is to tell you what I did with reasonable detail so that somebody skilled in the art could reproduce my results within the variability of the data and methods themselves.
The legitimate problem is that even I, with my data and memories of what I did before, can't necessarily run all of my old simulations again because libraries change, compilers change, and even languages change. So after a few years I'm at the same disadvantage as a stranger reading my papers. I'd have to rebuild the infrastructure to perform my simulations from scratch.
So, how much do you want the world to bet that your results are legit? This is the climate science problem, we are being asked to spend trillions, but it seems to be too difficult or somehow wrong to release the data and the code that justify this spending.
No, it's not "the climate science problem." For starters, you can't extrapolate from the HAD-CRUT to all of climate science. The folks who do the GISS-TEMP analysis tell you precisely how to replicate their analysis - all the code and data are publicly available. The source for a number of climate models are also available; same for many of the millennial scale climate reconstructions.
Does climate science face the same versioning and documentation problems as other fields? Of course. But climatologists are probably in better shape than genomics researchers, at least in regards to the amount of data they need to cope with.
In any case, we're going to have to spend trillions to continue extracting resources at our current rate (oil fields and power plants don't build themselves), so it's more a matter of prioritizing than spending.
Dr. Jay, another thing I would add to Ally's comment about global warming - even IF the GW scientists are doing something questionably repeatable, what the anti-GW people are doing is not science from the very start. They take their conclusion as correct - that there either isn't any, or it isn't man-caused, and then fit their data to support it. If you do this, you will ALWAYS come up with support - just reject anything NOT on the basis of identified problems, but on the basis that is doesn't support the conclusion.
Portability and Archivability - I didn't see either of these concepts mentioned. But these should be factored into the issue of scientific reproducibility of experiments. If at all possible, your peers should require no special platform-specific tools to reproduce your results. Likewise, you yourself should consider the likelihood that your own tools could become irrelevant in say 20 years. Plan accordingly.
Published data (input and output) should be in a plain-text "export" format. If source code is included, it should be clearly written and commented in a high-order procedural language, e.g. - Fortran, C, Perl. And "black box" applications and functions should definitely be avoided. But if such tools are unavoidable, the algorithmic assumptions should at least be clearly stated in the paper.
Unix and its built-in tool set will be around for a very long time, and should be your platform of choice. Avoid using object-oriented languages as your source code unless you eschew the libraries and methods that are the major rationale for OOP. Do your peers have those same libraries, and will you 20 years hence?
You wouldn't dream of archiving your most precious documents in MS Word, or you tax records in TurboTax format. In 20 years, both types would be unreadable on a future platform. Why treat scientific reproducibility any differently?