  We formulate tracking as an online decision-making process, where a tracking
agent must follow an object despite ambiguous image frames and a limited
computational budget. Crucially, the agent must decide where to look in the
upcoming frames, when to reinitialize because it believes the target has been
lost, and when to update its appearance model for the tracked object. Such
decisions are typically made heuristically. Instead, we propose to learn an
optimal decision-making policy by formulating tracking as a partially
observable decision-making process (POMDP). We learn policies with deep
reinforcement learning algorithms that need supervision (a reward signal) only
when the track has gone awry. We demonstrate that sparse rewards allow us to
quickly train on massive datasets, several orders of magnitude more than past
work. Interestingly, by treating the data source of Internet videos as
unlimited streams, we both learn and evaluate our trackers in a single, unified
computational stream.
