  Music Information Retrieval (MIR) research is increasingly leveraging
representation learning to obtain more compact, powerful music audio
representations for various downstream MIR tasks. However, current
representation evaluation methods are fragmented due to discrepancies in audio
and label preprocessing, downstream model and metric implementations, data
availability, and computational resources, often leading to inconsistent and
limited results. In this work, we introduce mir_ref, an MIR Representation
Evaluation Framework focused on seamless, transparent, local-first experiment
orchestration to support representation development. It features
implementations of a variety of components such as MIR datasets, tasks,
embedding models, and tools for result analysis and visualization, while
facilitating the implementation of custom components. To demonstrate its
utility, we use it to conduct an extensive evaluation of several embedding
models across various tasks and datasets, including evaluating their robustness
to various audio perturbations and the ease of extracting relevant information
from them.
