  The reproducibility of scientific findings are an important hallmark of
quality and integrity in research. The scientific method requires hypotheses to
be subjected to the most crucial tests, and for the results to be consistent
across independent trials. Therefore, a publication is expected to provide
sufficient information for an objective evaluation of its methods and claims.
This is particularly true for research supported by public funds, where
transparency of findings are a form of return on public investment.
Unfortunately, many publications fall short of this mark for various reasons,
including unavoidable ones such as intellectual property protection and
national security of the entity creating those findings. This is a particularly
important and documented problem in medical research, and in machine learning.
Fortunately for those seeking to overcome these difficulties, the internet
makes it easier to share experiments, and allows for crowd-sourced reverse
engineering. A case study of this capability in neural networks research is
presented in this paper. The significant success of reverse-engineering the
important accomplishments of DeepMind's Alpha Zero exemplifies the leverage
that can be achieved by a concerted effort to reproduce results.
