  Existing losses used in deep metric learning (DML) for image retrieval often
lead to highly non-uniform intra-class and inter-class representation
structures across test classes and data distributions. When combined with the
common practice of using a fixed threshold to declare a match, this gives rise
to significant performance variations in terms of false accept rate (FAR) and
false reject rate (FRR) across test classes and data distributions. We define
this issue in DML as threshold inconsistency. In real-world applications, such
inconsistency often complicates the threshold selection process when deploying
commercial image retrieval systems. To measure this inconsistency, we propose a
novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS)
that quantifies the variance in the operating characteristics across classes.
Using the OPIS metric, we find that achieving high accuracy levels in a DML
model does not automatically guarantee threshold consistency. In fact, our
investigation reveals a Pareto frontier in the high-accuracy regime, where
existing methods to improve accuracy often lead to degradation in threshold
consistency. To address this trade-off, we introduce the Threshold-Consistent
Margin (TCM) loss, a simple yet effective regularization technique that
promotes uniformity in representation structures across classes by selectively
penalizing hard sample pairs. Extensive experiments demonstrate TCM's
effectiveness in enhancing threshold consistency while preserving accuracy,
simplifying the threshold selection process in practical DML settings.
