  The field of bibliometrics, studying citations and behavior, is critical to
the discussion of reproducibility. Citations are one of the primary incentive
and reward systems for academic work, and so we desire to know if this
incentive rewards reproducible work. Yet to the best of our knowledge, only one
work has attempted to look at this combined space, concluding that
non-reproducible work is more highly cited. We show that answering this
question is more challenging than first proposed, and subtle issues can inhibit
a robust conclusion. To make inferences with more robust behavior, we propose a
hierarchical Bayesian model that incorporates the citation rate over time,
rather than the total number of citations after a fixed amount of time. In
doing so we show that, under current evidence the answer is more likely that
certain fields of study such as Medicine and Machine Learning (ML) do correlate
reproducible works with more citations, but other fields appear to have no
relationship. Further, we find that making code available and thoroughly
referencing prior works appear to also positively correlate with increased
citations. Our code and data can be found at
https://github.com/EdwardRaff/ReproducibleCitations .
