  How do the neural networks distinguish two images? It is of critical
importance to understand the matching mechanism of deep models for developing
reliable intelligent systems for many risky visual applications such as
surveillance and access control. However, most existing deep metric learning
methods match the images by comparing feature vectors, which ignores the
spatial structure of images and thus lacks interpretability. In this paper, we
present a deep interpretable metric learning (DIML) method for more transparent
embedding learning. Unlike conventional metric learning methods based on
feature vector comparison, we propose a structural matching strategy that
explicitly aligns the spatial embeddings by computing an optimal matching flow
between feature maps of the two images. Our method enables deep models to learn
metrics in a more human-friendly way, where the similarity of two images can be
decomposed to several part-wise similarities and their contributions to the
overall similarity. Our method is model-agnostic, which can be applied to
off-the-shelf backbone networks and metric learning methods. We evaluate our
method on three major benchmarks of deep metric learning including CUB200-2011,
Cars196, and Stanford Online Products, and achieve substantial improvements
over popular metric learning methods with better interpretability. Code is
available at https://github.com/wl-zhao/DIML
