  We present a predictor-corrector framework, called PicCoLO, that can
transform a first-order model-free reinforcement or imitation learning
algorithm into a new hybrid method that leverages predictive models to
accelerate policy learning. The new "PicCoLOed" algorithm optimizes a policy by
recursively repeating two steps: In the Prediction Step, the learner uses a
model to predict the unseen future gradient and then applies the predicted
estimate to update the policy; in the Correction Step, the learner runs the
updated policy in the environment, receives the true gradient, and then
corrects the policy using the gradient error. Unlike previous algorithms,
PicCoLO corrects for the mistakes of using imperfect predicted gradients and
hence does not suffer from model bias. The development of PicCoLO is made
possible by a novel reduction from predictable online learning to adversarial
online learning, which provides a systematic way to modify existing first-order
algorithms to achieve the optimal regret with respect to predictable
information. We show, in both theory and simulation, that the convergence rate
of several first-order model-free algorithms can be improved by PicCoLO.
