  We introduce a new framework that performs decision-making in reinforcement
learning (RL) as an iterative reasoning process. We model agent behavior as the
steady-state distribution of a parameterized reasoning Markov chain (RMC),
optimized with a new tractable estimate of the policy gradient. We perform
action selection by simulating the RMC for enough reasoning steps to approach
its steady-state distribution. We show our framework has several useful
properties that are inherently missing from traditional RL. For instance, it
allows agent behavior to approximate any continuous distribution over actions
by parameterizing the RMC with a simple Gaussian transition function. Moreover,
the number of reasoning steps to reach convergence can scale adaptively with
the difficulty of each action selection decision and can be accelerated by
re-using past solutions. Our resulting algorithm achieves state-of-the-art
performance in popular Mujoco and DeepMind Control benchmarks, both for
proprioceptive and pixel-based tasks.
