  Human similarity judgments are a powerful supervision signal for machine
learning applications based on techniques such as contrastive learning,
information retrieval, and model alignment, but classical methods for
collecting human similarity judgments are too expensive to be used at scale.
Recent methods propose using pre-trained deep neural networks (DNNs) to
approximate human similarity, but pre-trained DNNs may not be available for
certain domains (e.g., medical images, low-resource languages) and their
performance in approximating human similarity has not been extensively tested.
We conducted an evaluation of 611 pre-trained models across three domains --
images, audio, video -- and found that there is a large gap in performance
between human similarity judgments and pre-trained DNNs. To address this gap,
we propose a new class of similarity approximation methods based on language.
To collect the language data required by these new methods, we also developed
and validated a novel adaptive tag collection pipeline. We find that our
proposed language-based methods are significantly cheaper, in the number of
human judgments, than classical methods, but still improve performance over the
DNN-based methods. Finally, we also develop `stacked' methods that combine
language embeddings with DNN embeddings, and find that these consistently
provide the best approximations for human similarity across all three of our
modalities. Based on the results of this comprehensive study, we provide a
concise guide for researchers interested in collecting or approximating human
similarity data. To accompany this guide, we also release all of the similarity
and language data, a total of 206,339 human judgments, that we collected in our
experiments, along with a detailed breakdown of all modeling results.
