Top Tips to Make Your Research Irreproducible
% Some top tips for doing irreproducible research. %
ITT.tex
\documentclass[a4paper,11pt]{article}
\usepackage[top=1.8cm,bottom=1.8cm,left=1.8cm,right=1.8cm,asymmetric]{geometry}
\usepackage{url}
\usepackage{paralist}
\usepackage{authblk}
\usepackage[colorlinks=true,hyperfootnotes=true]{hyperref}

\title{Top Tips to Make Your Research Irreproducible}

% authors
\author[1]{Neil P. Chue Hong}
\author[2]{Tom Crick}
\author[3]{Ian P. Gent}
\author[4]{Lars Kotthoff}
\author[5]{Kenji Takeda}
% institutions
\affil[1]{Software Sustainability Institute, University of Edinburgh, UK}
\affil[2]{Department of Computing \& Information Systems, Cardiff Metropolitan University, UK}
\affil[3]{School of Computer Science, University of St Andrews, UK}
\affil[4]{Insight Centre for Data Analytics, University College Cork, Ireland}
\affil[5]{Microsoft Research, Cambridge, UK}
% emails
\affil[1]{\url{n.chuehong@software.ac.uk} ~~~~ \protect\url{http://www2.epcc.ed.ac.uk/\~neilc/} }
\affil[2]{\url{tcrick@cardiffmet.ac.uk} ~~~~ \protect\url{http://drtomcrick.com/}}
\affil[3]{\url{ian.gent@st-andrews.ac.uk} ~~~~ \protect\url{http://ian.gent}}
\affil[4]{\url{lars.kotthoff@insight-centre.org} ~~~~ \protect\url{http://4c.ucc.ie/\~larsko/}}
\affil[5]{\url{kenji.takeda@microsoft.com} ~~~~ \protect\url{http://research.microsoft.com/en-us/people/kenjitak/}}

\renewcommand\Authands{ and }
\def\UrlBreaks{\do\/\do-}

\date{1 April 2015}

\begin{document}
\maketitle

% c.800 words?
% General idea
% Come up with a paper on Irreproducibility Top Tips
% Use specific personal examples from the authors to back each tip up
% \begin{abstract}
% Some top tips for doing irreproducible research.
% \end{abstract}

% from Recomputability 2014 workshop
% \section{Workshop List}
% This is the list we came up with at the workshop
% \begin{enumerate}
% \item implement everything yourself / don't reuse existing, well tested code wherever possible
% \item  don't share anything
% \ don't make it clear what irreproducibility means in your context / domain
% \item don't make it clear where your method / approximations cannot be / should not be used
% \item don't publish your source code, or your implementation
% \item  define custom benchmarks
% \item  don't describe your experimental setup
% \item  give a vague methodology
% \item  data formats are boring
% \item  use magic numbers everywhere
% \item  miss out key parts / fudgy workflow
% \item  don't share the raw data you used to build everything
% \item  define your own data formats, preferably in binary, mont human or machine readable
% \item don't use source code control software for either code or document source like latex
% % IPG can think of two examples where we lost paper latex files and definitely/probably lost journal papers as result.
% \item it's only necessary that your code build and run on one physical computer on one date in history. 
% \end{enumerate}

% also from Recomputability 2014
% \section{Lars distillation}
% Need to distil those down to probably not more than 5 points -- here's Lars'
% attempt:
% \begin{enumerate}
% \item Pseudo-code is a great way of communicating ideas quickly and clearly.
% \item Don't share your implementation, you only make it easier for other people
% to scoop your ideas.
% \item People are interested in the science, not the experimental setup, so don't
% describe it.
% \item You're \emph{the} expert in the domain, only you can define what
% algorithms and data to run experiments with. (TC: Thus, you should
% create your own benchmarks and use those)
% \item Any limitations of the method and approximation will be obvious to the
% careful reader, no need to waste space on making them explicit (TC:
% quote Fermat, save space, only write stuff in the margins and obliquely
% refer to amazing proofs).
% \end{enumerate}

%\section{Ian's attempt at an April Fools' version}
% We have noticed a number of guides being published on how to make
% research reproducible. However, we have seen very little published on
% how to make research \emph{irreproducible}.  Irreproducibility is the
% default setting for all of science, and irreproducible research is
% particularly common in computational science.  The study of making
% your work irreproducible without reviewers complaining is a much
% neglected area. We feel therefore that by giving our top tips on
% irreproducibility, we will be filling a much needed gap in the
% literature.  %\footnote{Filling a much needed gap is a standard joke.}
% By following our tips you can ensure that if you work is wrong, nobody
% can find it out, while if it's right you will make everyone else do
% significantly more work than you to build on it. In either case you
% are the beneficiary.

% Our most vital tip is deceptively but beautifully simple. To ensure
% irreproducibility of your work, make sure that you cannot reproduce it
% yourself. If you were able to reproduce it, there would always be the
% danger of somebody else being able to do exactly the same as you. Much
% else follows from this. For example, complete confidence in your own
% inability to reproduce work saves tedious time revising your work on
% advice from reviewers: if you can't browbeat the editor into accepting
% it as is, you can resubmit elsewhere. A major advantage of this key
% insight is that no strict discipline is required to ensure
% self-irreproducibility: in our experience, irreproducibility can
% happily occur after only the tiniest amount of carelessness is
% necessary at one of any number of stages.

% There is an unfortunate convention in science that research should
% pretend to be reproducible. The remainder of our top tips will help
% you salve the conscience of reviewers still bound by this fussy
% conventionality, enabling them to enthusiastically recommend
% acceptance of your irreproducible work.

% \begin{enumerate}
% \item Pseudo-code is a great way of communicating ideas quickly and
% clearly while giving readers no chance to understand the subtle
% implementation details that make it work.
% \item Don't share your implementation. Doing so only makes it easier
% for other people to scoop your ideas, understand why your code
% actually works instead of why you say it does, or worst of all to
% understand that your code doesn't work at all.
% \item People are interested in the science, not the experimental
% setup, so don't describe it.  If necessary, camouflage this absence
% with brief details of insignificant aspects of your methodology.
% \item You're \emph{the} expert in the domain, only you can define what
% algorithms and data to run experiments with. In the unhappy
% circumstance that your methods do not do well on standard benchmarks,
% you should create your own benchmarks and use those but preferably not
% make them available to others.
% \item Any limitations of your methods or proofs will be obvious to the
% careful reader, so there is no need to waste space on making them
% explicit.\footnote{Space saved in this way can be used to cite the
% critical papers in the field, i.e. those papers that will increment
% potential reviewers' h-index.}  However much work it takes colleagues
% to fill in the gaps, you will still get the credit if you just say you
% have amazing experiments or proofs.  After all, "Fermat's Last
% Theorem" is still known by that name even though somebody else proved
% it.
% \end{enumerate}

% We leave you with a simple postulate: that an experiment that is
% irreproducible is exactly equivalent to an experiment that was never
% carried out at all. The happy consequences of this postulate for
% experts in irreproducibility will be published elsewhere.

% It has not escaped our notice that this specific equivalence we have postulated immediately suggests a possible mechanism for greatly increasing the amount of scientific material.
% cf. Crick and Watson DNA paper: "It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material."



% TC's attempt at an April Fool's version
We have noticed (and contributed to) a number of manifestos, guides
and top tips on how to make research
reproducible~\cite{prlic+procter:2012,sandve-et-al:2013,gent:2013,joppa-et-al:2013,gent_recomputation.org_2014,wilson-et-al:2014,goble:2014,crick-et-al_wssspe2,crick-et-al_recomp2014,stodden+miguez:2014};
however, we have seen very little published on how to make research
\emph{irreproducible}.

Irreproducibility is the default setting for all of science, and
irreproducible research is particularly common across the
computational sciences. The study of making your work irreproducible
without reviewers complaining is a much neglected area; we feel
therefore that by encapsulating some of our top
tips\footnote{{\emph{N.B.}} We are by no means claiming this is an
exhaustive list for making your research irreproducible...} on
irreproducibility, we will be filling a much-needed gap in the domain
literature. By following our starter tips, you can ensure that if your
work is wrong, nobody will be able to check it; if it is correct, you
will make everyone else do disproportionately more work than you to
build upon it. In either case you are the beneficiary.

It is an unfortunate convention of science that research should
pretend to be reproducible; our top tips will help you salve the
conscience of certain reviewers still bound by this fussy
conventionality, enabling them to enthusiastically recommend
acceptance of your irreproducible work.

\begin{enumerate}
\item {\textbf{Think ``Big Picture''.}} People are interested in the
science, not the dull experimental setup, so don't describe it. If
necessary, camouflage this absence with brief, high-level details of
insignificant aspects of your methodology.
\item {\textbf{Be abstract.}} Pseudo-code is a great way of
communicating ideas quickly and clearly while giving readers no chance
to understand the subtle implementation details (particularly the
custom toolchains and manual interventions) that actually make it work.
\item {\textbf{Short and sweet.}} Any limitations of your
methods or proofs will be obvious to the careful reader, so there is
no need to waste space on making them explicit\footnote{Space saved in
this way can be used to cite the critical papers in the field,
i.e. those papers that will inflate your own (as well as potential
reviewers') h-index.}. However much work it takes colleagues to fill
in the gaps, you will still get the credit if you just say you have
amazing experiments or proofs (with a hat-tip to Pierre de Fermat:
``{\emph{Cuius rei demonstrationem mirabilem sane detexi hanc marginis
exiguitas non caperet.}}'').
\item {\textbf{The deficit model.}} You're \emph{the} expert in the
domain, only you can define what algorithms and data to run
experiments with. In the unhappy circumstance that your methods do not
do well on community curated benchmarks, you should create your own
bespoke benchmarks and use those (and preferably not make them
available to others).
\item {\textbf{Don't share.}} Doing so only makes it easier for other
people to scoop your research ideas, understand how your code actually
works\footnote{An exemplary example:
\url{http://www.phdcomics.com/comics.php?f=1689}} instead of why you
say it does, or worst of all to understand that your code doesn't
actually work at all.
\end{enumerate}

However, our most important tip is deceptively but beautifully simple:
{\textbf{to ensure your work is irreproducible, make sure that you
cannot reproduce it yourself}}. If you were able to reproduce it,
there would always be the danger of somebody else being able to do
exactly the same as you. Much else follows from this; for example,
complete confidence in your own inability to reproduce work saves
tedious time revising your work on advice from reviewers: if you are
unable to browbeat the editor into accepting it as is, you can always
resubmit elsewhere. A major advantage of this key insight is that no
strict discipline is required to ensure self-irreproducibility: in our
experience, irreproducibility can happily occur after only the tiniest
amount of carelessness at one of any number of stages.

We make a simple conjecture: {\textbf{an experiment that is
irreproducible is exactly equivalent to an experiment that was never
carried out at all}}. The happy consequences of this conjecture for
experts in irreproducibility will be published elsewhere, with
extremely impressive experimental support.\\

\noindent We close with a mantra for scientists interested in irreproducibility:
\begin{quote}
\emph{After
Publishing
Research,
Irreproducibility 
Lets
False
Observations
Obtain
Longevity!}
\end{quote}

% \section{David's stuff}

% \begin{itemize}
% 	\item If you use tools that are accepted in the field, such as
% 		simulation packages, fix bugs in them during the course of your
% 		work. Assume that these are obvious errors in functionality and
% 		could not possibly affect your results, so you should maintain a
% 		hazy memory of what the bugs were and the fixes entailed.

% 	\item Use poor experimental design, especially by not keeping
% 		records of PRNG seeds and using few replications (preferably
% 		one). This increases the potential uniqueness of your results.
% 		This statistical tomfoolery may lead to prominent features in
% 		any graphs you present, so either discuss them heavily or ignore
% 		them entirely so they are left as a pregnant mystery to the
% 		reader. This approach is illustrated by Wong, Evans, and
% 		Kock~\cite{Wong2000a}.

% \end{itemize}

% \section*{Notes}

% Other questions that came up

% Are the trustable minimal information for recomputability  different or overlapping for different domains?
% \begin{itemize}
% \item in AI alone very different
% \item require publishing of algorithm and implementation
% \item what stepping of the memory controller in systems domain
% \end{itemize}

% Is there a Dublin Core - or a set of principles - that we can agree
% for recomputability?

% What about a paper which is essentially theoretical but complex, and
% an implementation is included to show that it is possible to implement
% it. Should that implementation be reproducible? Arguably not.



% bib
\bibliographystyle{unsrt}
\bibliography{ITT}

\end{document}

ITT.bbl
\begin{thebibliography}{10}

\bibitem{prlic+procter:2012}
Andreas Prli\'{c} and James~B. Procter.
\newblock {Ten Simple Rules for the Open Development of Scientific Software}.
\newblock {\em {PLoS Computational Biology}}, 12(8), 2012.

\bibitem{sandve-et-al:2013}
{Geir Kjetil} Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig.
\newblock {Ten Simple Rules for Reproducible Computational Research}.
\newblock {\em {PLoS Computational Biology}}, 9(10), 2013.

\bibitem{gent:2013}
Ian~P. Gent.
\newblock {The Recomputation Manifesto}.
\newblock Available from: \url{http://arxiv.org/abs/1304.3674}, April 2013.

\bibitem{joppa-et-al:2013}
Lucas~N. Joppa, Greg McInerny, Richard Harper, Lara Salido, Kenji Takeda,
  Kenton O'Hara, David Gavaghan, and Stephen Emmott.
\newblock {Troubling Trends in Scientific Software Use}.
\newblock {\em Science}, 340(6134):814--815, 2013.

\bibitem{gent_recomputation.org_2014}
Ian~P. Gent and Lars Kotthoff.
\newblock Recomputation.org: Experience of its first year and lessons learned.
\newblock In {\em Recomputability 2014}, December 2014.

\bibitem{wilson-et-al:2014}
Greg Wilson, D.~A. Aruliah, {C. Titus} Brown, Neil~P. {Chue Hong}, Matt Davis,
  Richard~T. Guy, Steven H.~D. Haddock, Kathryn~D. Huff, Ian~M. Mitchell,
  Mark~D. Plumbley, Ben Waugh, Ethan~P. White, and Paul Wilson.
\newblock {Best Practices for Scientific Computing}.
\newblock {\em {PLoS Biology}}, 12(1), 2014.

\bibitem{goble:2014}
Carole Goble.
\newblock {Better Software, Better Research}.
\newblock {\em {IEEE Internet Computing}}, 18(5):4--8, 2014.

\bibitem{crick-et-al_wssspe2}
Tom Crick, Benjamin~A. Hall, and Samin Ishtiaq.
\newblock {``Can I Implement Your Algorithm?'': A Model for Reproducible
  Research Software}.
\newblock In {\em {Proceedings of 2nd International Workshop on Sustainable
  Software for Science: Practice and Experiences (WSSSPE2)}}, 2014.

\bibitem{crick-et-al_recomp2014}
Tom Crick, Benjamin~A. Hall, Samin Ishtiaq, and Kenji Takeda.
\newblock {``Share and Enjoy'': Publishing Useful (and Usable) Scientific
  Models}.
\newblock In {\em {Proceedings of 1st International Workshop on
  Recomputability}}, 2014.

\bibitem{stodden+miguez:2014}
Victoria Stodden and Sheila Miguez.
\newblock {Best Practices for Computational Science: Software Infrastructure
  and Environments for Reproducible and Extensible Research}.
\newblock {\em {Journal of Open Research Software}}, 2(1):1--6, 2014.

\end{thebibliography}
