  We propose a novel method for template matching in unconstrained
environments. Its essence is the Best-Buddies Similarity (BBS), a useful,
robust, and parameter-free similarity measure between two sets of points. BBS
is based on counting the number of Best-Buddies Pairs (BBPs)--pairs of points
in source and target sets, where each point is the nearest neighbor of the
other. BBS has several key features that make it robust against complex
geometric deformations and high levels of outliers, such as those arising from
background clutter and occlusions. We study these properties, provide a
statistical analysis that justifies them, and demonstrate the consistent
success of BBS on a challenging real-world dataset while using different types
of features.
