  Embedders play a central role in machine learning, projecting any object into
numerical representations that can, in turn, be leveraged to perform various
downstream tasks. The evaluation of embedding models typically depends on
domain-specific empirical approaches utilizing downstream tasks, primarily
because of the lack of a standardized framework for comparison. However,
acquiring adequately large and representative datasets for conducting these
assessments is not always viable and can prove to be prohibitively expensive
and time-consuming. In this paper, we present a unified approach to evaluate
embedders. First, we establish theoretical foundations for comparing embedding
models, drawing upon the concepts of sufficiency and informativeness. We then
leverage these concepts to devise a tractable comparison criterion (information
sufficiency), leading to a task-agnostic and self-supervised ranking procedure.
We demonstrate experimentally that our approach aligns closely with the
capability of embedding models to facilitate various downstream tasks in both
natural language processing and molecular biology. This effectively offers
practitioners a valuable tool for prioritizing model trials.
