  In recent years, ML researchers have wrestled with defining and improving
machine learning (ML) benchmarks and datasets. In parallel, some have trained a
critical lens on the ethics of dataset creation and ML research. In this
position paper, we highlight the entanglement of ethics with seemingly
``technical'' or ``scientific'' decisions about the design of ML benchmarks.
Our starting point is the existence of multiple overlooked structural
similarities between human intelligence benchmarks and ML benchmarks. Both
types of benchmarks set standards for describing, evaluating, and comparing
performance on tasks relevant to intelligence -- standards that many scholars
of human intelligence have long recognized as value-laden. We use perspectives
from feminist philosophy of science on IQ benchmarks and thick concepts in
social science to argue that values need to be considered and documented when
creating ML benchmarks. It is neither possible nor desirable to avoid this
choice by creating value-neutral benchmarks. Finally, we outline practical
recommendations for ML benchmark research ethics and ethics review.
