  Model selection for a given target task can be costly, as it may entail
extensive annotation of the quality of outputs of different models. We
introduce DiffUse, an efficient method to make an informed decision between
candidate text generation models based on preference annotations. DiffUse
reduces the required amount of annotations, thus saving valuable time and
resources in performing evaluation. DiffUse intelligently selects instances by
clustering embeddings that represent the semantic differences between model
outputs. Thus, it is able to identify a subset of examples that are more
informative for preference decisions. Our method is model-agnostic, and can be
applied to any text generation model for selecting between models, prompts and
configurations. Moreover, we propose a practical iterative approach for
dynamically determining how many instances to annotate. In a series of
experiments over hundreds of model pairs, we demonstrate that DiffUse can
dramatically reduce the required number of annotations -- by up to 75% -- while
maintaining high evaluation reliability.
