  In many real-world applications of reinforcement learning (RL), interactions
with the environment are limited due to cost or feasibility. This presents a
challenge to traditional RL algorithms since the max-return objective involves
an expectation over on-policy samples. We introduce a new formulation of
max-return optimization that allows the problem to be re-expressed by an
expectation over an arbitrary behavior-agnostic and off-policy data
distribution. We first derive this result by considering a regularized version
of the dual max-return objective before extending our findings to unregularized
objectives through the use of a Lagrangian formulation of the linear
programming characterization of Q-values. We show that, if auxiliary dual
variables of the objective are optimized, then the gradient of the off-policy
objective is exactly the on-policy policy gradient, without any use of
importance weighting. In addition to revealing the appealing theoretical
properties of this approach, we also show that it delivers good practical
performance.
