  Recent successful deep reinforcement learning algorithms, such as Trust
Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are
fundamentally variations of conservative policy iteration (CPI). These
algorithms iterate policy evaluation followed by a softened policy improvement
step. As so, they are naturally on-policy. In this paper, we propose to combine
(any kind of) soft greediness with Modified Policy Iteration (MPI). The
proposed abstract framework applies repeatedly: (i) a partial policy evaluation
step that allows off-policy learning and (ii) any softened greedy step. Our
contribution can be seen as a new generic tool for the deep reinforcement
learning toolbox. As a proof of concept, we instantiate this framework with the
PPO greediness. Comparison to the original PPO shows that our algorithm is much
more sample efficient. We also show that it is competitive with the
state-of-art off-policy algorithm Soft Actor Critic (SAC).
