  Off-policy policy optimization is a challenging problem in reinforcement
learning (RL). The algorithms designed for this problem often suffer from high
variance in their estimators, which results in poor sample efficiency, and have
issues with convergence. A few variance-reduced on-policy policy gradient
algorithms have been recently proposed that use methods from stochastic
optimization to reduce the variance of the gradient estimate in the REINFORCE
algorithm. However, these algorithms are not designed for the off-policy
setting and are memory-inefficient, since they need to collect and store a
large ``reference'' batch of samples from time to time. To achieve
variance-reduced off-policy-stable policy optimization, we propose an algorithm
family that is memory-efficient, stochastically variance-reduced, and capable
of learning from off-policy samples. Empirical studies validate the
effectiveness of the proposed approaches.
