  Object tracking is an essential problem in computer vision that has been
researched for several decades. One of the main challenges in tracking is to
adapt to object appearance changes over time and avoiding drifting to
background clutter. We address this challenge by proposing a deep neural
network composed of different parts, which functions as a society of tracking
parts. They work in conjunction according to a certain policy and learn from
each other in a robust manner, using co-occurrence constraints that ensure
robust inference and learning. From a structural point of view, our network is
composed of two main pathways. One pathway is more conservative. It carefully
monitors a large set of simple tracker parts learned as linear filters over
deep feature activation maps. It assigns the parts different roles. It promotes
the reliable ones and removes the inconsistent ones. We learn these filters
simultaneously in an efficient way, with a single closed-form formulation, for
which we propose novel theoretical properties. The second pathway is more
progressive. It is learned completely online and thus it is able to better
model object appearance changes. In order to adapt in a robust manner, it is
learned only on highly confident frames, which are decided using co-occurrences
with the first pathway. Thus, our system has the full benefit of two main
approaches in tracking. The larger set of simpler filter parts offers
robustness, while the full deep network learned online provides adaptability to
change. As shown in the experimental section, our approach achieves state of
the art performance on the challenging VOT17 benchmark, outperforming the
published methods both on the general EAO metric and in the number of fails, by
a significant margin.
