  In this article I propose an approach for defining replicability for
prediction rules. Motivated by a recent NAS report, I start from the
perspective that replicability is obtaining consistent results across studies
suitable to address the same prediction question, each of which has obtained
its own data. I then discuss concept and issues in defining key elements of
this statement. I focus specifically on the meaning of "consistent results" in
typical utilization contexts, and propose a multi-agent framework for defining
replicability, in which agents are neither partners nor adversaries. I recover
some of the prevalent practical approaches as special cases. I hope to provide
guidance for a more systematic assessment of replicability in machine learning.
