  Forming a reliable judgement of a machine learning (ML) model's
appropriateness for an application ecosystem is critical for its responsible
use, and requires considering a broad range of factors including harms,
benefits, and responsibilities. In practice, however, evaluations of ML models
frequently focus on only a narrow range of decontextualized predictive
behaviours. We examine the evaluation gaps between the idealized breadth of
evaluation concerns and the observed narrow focus of actual evaluations.
Through an empirical study of papers from recent high-profile conferences in
the Computer Vision and Natural Language Processing communities, we demonstrate
a general focus on a handful of evaluation methods. By considering the metrics
and test data distributions used in these methods, we draw attention to which
properties of models are centered in the field, revealing the properties that
are frequently neglected or sidelined during evaluation. By studying these
properties, we demonstrate the machine learning discipline's implicit
assumption of a range of commitments which have normative impacts; these
include commitments to consequentialism, abstractability from context, the
quantifiability of impacts, the limited role of model inputs in evaluation, and
the equivalence of different failure modes. Shedding light on these assumptions
enables us to question their appropriateness for ML system contexts, pointing
the way towards more contextualized evaluation methodologies for robustly
examining the trustworthiness of ML models
