  The replicability crisis in the social, behavioral, and data sciences has led
to the formulation of algorithm frameworks for replicability -- i.e., a
requirement that an algorithm produce identical outputs (with high probability)
when run on two different samples from the same underlying distribution. While
still in its infancy, provably replicable algorithms have been developed for
many fundamental tasks in machine learning and statistics, including
statistical query learning, the heavy hitters problem, and distribution
testing. In this work we initiate the study of replicable reinforcement
learning, providing a provably replicable algorithm for parallel value
iteration, and a provably replicable version of R-max in the episodic setting.
These are the first formal replicability results for control problems, which
present different challenges for replication than batch learning settings.
