  Recently, vision-based control has gained traction by leveraging the power of
machine learning. In this work, we couple a model predictive control (MPC)
framework to a visual pipeline. We introduce deep optical flow (DOF) dynamics,
which is a combination of optical flow and robot dynamics. Using the DOF
dynamics, MPC explicitly incorporates the predicted movement of relevant pixels
into the planned trajectory of a robot. Our implementation of DOF is
memory-efficient, data-efficient, and computationally cheap so that it can be
computed in real-time for use in an MPC framework. The suggested Pixel Model
Predictive Control (PixelMPC) algorithm controls the robot to accomplish a
high-speed racing task while maintaining visibility of the important features
(gates). This improves the reliability of vision-based estimators for
localization and can eventually lead to safe autonomous flight. The proposed
algorithm is tested in a photorealistic simulation with a high-speed drone
racing task.
