  An important problem that arises in reinforcement learning and Monte Carlo
methods is estimating quantities defined by the stationary distribution of a
Markov chain. In many real-world applications, access to the underlying
transition operator is limited to a fixed set of data that has already been
collected, without additional interaction with the environment being available.
We show that consistent estimation remains possible in this challenging
scenario, and that effective estimation can still be achieved in important
applications. Our approach is based on estimating a ratio that corrects for the
discrepancy between the stationary and empirical distributions, derived from
fundamental properties of the stationary distribution, and exploiting
constraint reformulations based on variational divergence minimization. The
resulting algorithm, GenDICE, is straightforward and effective. We prove its
consistency under general conditions, provide an error analysis, and
demonstrate strong empirical performance on benchmark problems, including
off-line PageRank and off-policy policy evaluation.
