  Reinforcement learning (RL) in low-data and risk-sensitive domains requires
performant and flexible deployment policies that can readily incorporate
constraints during deployment. One such class of policies are the
semi-parametric H-step lookahead policies, which select actions using
trajectory optimization over a dynamics model for a fixed horizon with a
terminal value function. In this work, we investigate a novel instantiation of
H-step lookahead with a learned model and a terminal value function learned by
a model-free off-policy algorithm, named Learning Off-Policy with Online
Planning (LOOP). We provide a theoretical analysis of this method, suggesting a
tradeoff between model errors and value function errors and empirically
demonstrate this tradeoff to be beneficial in deep reinforcement learning.
Furthermore, we identify the "Actor Divergence" issue in this framework and
propose Actor Regularized Control (ARC), a modified trajectory optimization
procedure. We evaluate our method on a set of robotic tasks for Offline and
Online RL and demonstrate improved performance. We also show the flexibility of
LOOP to incorporate safety constraints during deployment with a set of
navigation environments. We demonstrate that LOOP is a desirable framework for
robotics applications based on its strong performance in various important RL
settings. Project video and details can be found at
https://hari-sikchi.github.io/loop .
