  As reproducibility becomes a greater concern, conferences have largely
converged to a strategy of asking reviewers to indicate whether code was
attached to a submission. This is part of a larger trend of taking action based
on assumed ideals, without studying if those actions will yield the desired
outcome. Our argument is that this focus on code for replication is misguided
if we want to improve the state of reproducible research. This focus can be
harmful -- we should not force code to be submitted. There is a lack of
evidence for effective actions taken by conferences to encourage and reward
reproducibility. We argue that venues must take more action to advance
reproducible machine learning research today.
