  In this paper, we propose a novel matching based tracker by investigating the
relationship between template matching and the recent popular correlation
filter based trackers (CFTs). Compared to the correlation operation in CFTs, a
sophisticated similarity metric termed "mutual buddies similarity" (MBS) is
proposed to exploit the relationship of multiple reciprocal nearest neighbors
for target matching. By doing so, our tracker obtains powerful discriminative
ability on distinguishing target and background as demonstrated by both
empirical and theoretical analyses. Besides, instead of utilizing single
template with the improper updating scheme in CFTs, we design a novel online
template updating strategy named "memory filtering" (MF), which aims to select
a certain amount of representative and reliable tracking results in history to
construct the current stable and expressive template set. This scheme is
beneficial for the proposed tracker to comprehensively "understand" the target
appearance variations, "recall" some stable results. Both qualitative and
quantitative evaluations on two benchmarks suggest that the proposed tracking
method performs favorably against some recently developed CFTs and other
competitive trackers.
