  Visual similarities discovery (VSD) is an important task with broad
e-commerce applications. Given an image of a certain object, the goal of VSD is
to retrieve images of different objects with high perceptual visual similarity.
Although being a highly addressed problem, the evaluation of proposed methods
for VSD is often based on a proxy of an identification-retrieval task,
evaluating the ability of a model to retrieve different images of the same
object. We posit that evaluating VSD methods based on identification tasks is
limited, and faithful evaluation must rely on expert annotations. In this
paper, we introduce the first large-scale fashion visual similarity benchmark
dataset, consisting of more than 110K expert-annotated image pairs. Besides
this major contribution, we share insight from the challenges we faced while
curating this dataset. Based on these insights, we propose a novel and
efficient labeling procedure that can be applied to any dataset. Our analysis
examines its limitations and inductive biases, and based on these findings, we
propose metrics to mitigate those limitations. Though our primary focus lies on
visual similarity, the methodologies we present have broader applications for
discovering and evaluating perceptual similarity across various domains.
