  In this paper we introduce a fully end-to-end approach for visual tracking in
videos that learns to predict the bounding box locations of a target object at
every frame. An important insight is that the tracking problem can be
considered as a sequential decision-making process and historical semantics
encode highly relevant information for future decisions. Based on this
intuition, we formulate our model as a recurrent convolutional neural network
agent that interacts with a video overtime, and our model can be trained with
reinforcement learning (RL) algorithms to learn good tracking policies that pay
attention to continuous, inter-frame correlation and maximize tracking
performance in the long run. The proposed tracking algorithm achieves
state-of-the-art performance in an existing tracking benchmark and operates at
frame-rates faster than real-time. To the best of our knowledge, our tracker is
the first neural-network tracker that combines convolutional and recurrent
networks with RL algorithms.
