  We develop an algorithm that combines model-based and model-free methods for
solving a nonlinear optimal control problem with a quadratic cost in which the
system model is given by a linear state-space model with a small additive
nonlinear perturbation. We decompose the cost into a sum of two functions, one
having an explicit form obtained from the approximate linear model, the other
being a black-box model representing the unknown modeling error. The
decomposition allows us to formulate the problem as a composite optimization
problem. To solve the optimization problem, our algorithm performs gradient
descent using the gradient obtained from the approximate linear model until
backtracking line search fails, upon which the model-based gradient is compared
with the exact gradient obtained from a model-free algorithm. The difference
between the model gradient and the exact gradient is then used for compensating
future gradient-based updates. Our algorithm is shown to decrease the number of
function evaluations compared with traditional model-free methods both in
theory and in practice.
