  Visual object tracking is a fundamental and time-critical vision task. Recent
years have seen many shallow tracking methods based on real-time pixel-based
correlation filters, as well as deep methods that have top performance but need
a high-end GPU. In this paper, we learn to improve the speed of deep trackers
without losing accuracy. Our fundamental insight is to take an adaptive
approach, where easy frames are processed with cheap features (such as pixel
values), while challenging frames are processed with invariant but expensive
deep features. We formulate the adaptive tracking problem as a decision-making
process, and learn an agent to decide whether to locate objects with high
confidence on an early layer, or continue processing subsequent layers of a
network. This significantly reduces the feed-forward cost for easy frames with
distinct or slow-moving objects. We train the agent offline in a reinforcement
learning fashion, and further demonstrate that learning all deep layers (so as
to provide good features for adaptive tracking) can lead to near real-time
average tracking speed of 23 fps on a single CPU while achieving
state-of-the-art performance. Perhaps most tellingly, our approach provides a
100X speedup for almost 50% of the time, indicating the power of an adaptive
approach.
