  Markov reward processes (MRPs) are used to model stochastic phenomena arising
in operations research, control engineering, robotics, and artificial
intelligence, as well as communication and transportation networks. In many of
these cases, such as in the policy evaluation problem encountered in
reinforcement learning, the goal is to estimate the long-term value function of
such a process without access to the underlying population transition and
reward functions. Working with samples generated under the synchronous model,
we study the problem of estimating the value function of an infinite-horizon,
discounted MRP on finitely many states in the $\ell_\infty$-norm. We analyze
both the standard plug-in approach to this problem and a more robust variant,
and establish non-asymptotic bounds that depend on the (unknown) problem
instance, as well as data-dependent bounds that can be evaluated based on the
observations of state-transitions and rewards. We show that these approaches
are minimax-optimal up to constant factors over natural sub-classes of MRPs.
Our analysis makes use of a leave-one-out decoupling argument tailored to the
policy evaluation problem, one which may be of independent interest.
