  Recommender Systems today are still mostly evaluated in terms of accuracy,
with other aspects beyond the immediate relevance of recommendations, such as
diversity, long-term user retention and fairness, often taking a back seat.
Moreover, reconciling multiple performance perspectives is by definition
indeterminate, presenting a stumbling block to those in the pursuit of rounded
evaluation of Recommender Systems. EvalRS 2022 -- a data challenge designed
around Multi-Objective Evaluation -- was a first practical endeavour, providing
many insights into the requirements and challenges of balancing multiple
objectives in evaluation. In this work, we reflect on EvalRS 2022 and expound
upon crucial learnings to formulate a first-principles approach toward
Multi-Objective model selection, and outline a set of guidelines for carrying
out a Multi-Objective Evaluation challenge, with potential applicability to the
problem of rounded evaluation of competing models in real-world deployments.
