  Recent advances in visual tracking are based on siamese feature extractors
and template matching. For this category of trackers, latest research focuses
on better feature embeddings and similarity measures. In this work, we focus on
building holistic object representations for tracking. We propose a framework
that is designed to be used on top of previous trackers without any need for
further training of the siamese network. The framework leverages the idea of
obtaining additional object templates during the tracking process. Since the
number of stored templates is limited, our method only keeps the most diverse
ones. We achieve this by providing a new diversity measure in the space of
siamese features. The obtained representation contains information beyond the
ground truth object location provided to the system. It is then useful for
tracking itself but also for further tasks which require a visual understanding
of objects. Strong empirical results on tracking benchmarks indicate that our
method can improve the performance and robustness of the underlying trackers
while barely reducing their speed. In addition, our method is able to match
current state-of-the-art results, while using a simpler and older network
architecture and running three times faster.
