  Fully convolutional deep correlation networks are integral components of
state-of the-art approaches to single object visual tracking. It is commonly
assumed that these networks perform tracking by detection by matching features
of the object instance with features of the entire frame. Strong architectural
priors and conditioning on the object representation is thought to encourage
this tracking strategy. Despite these strong priors, we show that deep trackers
often default to tracking by saliency detection - without relying on the object
instance representation. Our analysis shows that despite being a useful prior,
salience detection can prevent the emergence of more robust tracking strategies
in deep networks. This leads us to introduce an auxiliary detection task that
encourages more discriminative object representations that improve tracking
performance.
