  Classifying the confusing samples in the course of RGBT tracking is a quite
challenging problem, which hasn't got satisfied solution. Existing methods only
focus on enlarging the boundary between positive and negative samples, however,
the structured information of samples might be harmed, e.g., confusing positive
samples are closer to the anchor than normal positive samples.To handle this
problem, we propose a novel Multi-Modal Multi-Margin Metric Learning framework,
named M$^5$L for RGBT tracking in this paper. In particular, we design a
multi-margin structured loss to distinguish the confusing samples which play a
most critical role in tracking performance boosting. To alleviate this problem,
we additionally enlarge the boundaries between confusing positive samples and
normal ones, between confusing negative samples and normal ones with predefined
margins, by exploiting the structured information of all samples in each
modality.Moreover, a cross-modality constraint is employed to reduce the
difference between modalities and push positive samples closer to the anchor
than negative ones from two modalities.In addition, to achieve quality-aware
RGB and thermal feature fusion, we introduce the modality attentions and learn
them using a feature fusion module in our network. Extensive experiments on
large-scale datasets testify that our framework clearly improves the tracking
performance and outperforms the state-of-the-art RGBT trackers.
