  An open research question in robotics is how to combine the benefits of
model-free reinforcement learning (RL) - known for its strong task performance
and flexibility in optimizing general reward formulations - with the robustness
and online replanning capabilities of model predictive control (MPC). This
paper provides an answer by introducing a new framework called Actor-Critic
Model Predictive Control. The key idea is to embed a differentiable MPC within
an actor-critic RL framework. The proposed approach leverages the short-term
predictive optimization capabilities of MPC with the exploratory and end-to-end
training properties of RL. The resulting policy effectively manages both
short-term decisions through the MPC-based actor and long-term prediction via
the critic network, unifying the benefits of both model-based control and
end-to-end learning. We validate our method in both simulation and the real
world with a quadcopter platform across various high-level tasks. We show that
the proposed architecture can achieve real-time control performance, learn
complex behaviors via trial and error, and retain the predictive properties of
the MPC to better handle out of distribution behaviour.
