  How should we decide which fairness criteria or definitions to adopt in
machine learning systems? To answer this question, we must study the fairness
preferences of actual users of machine learning systems. Stringent parity
constraints on treatment or impact can come with trade-offs, and may not even
be preferred by the social groups in question (Zafar et al., 2017). Thus it
might be beneficial to elicit what the group's preferences are, rather than
rely on a priori defined mathematical fairness constraints. Simply asking for
self-reported rankings of users is challenging because research has shown that
there are often gaps between people's stated and actual preferences(Bernheim et
al., 2013).
  This paper outlines a research program and experimental designs for
investigating these questions. Participants in the experiments are invited to
perform a set of tasks in exchange for a base payment--they are told upfront
that they may receive a bonus later on, and the bonus could depend on some
combination of output quantity and quality. The same group of workers then
votes on a bonus payment structure, to elicit preferences. The voting is
hypothetical (not tied to an outcome) for half the group and actual (tied to
the actual payment outcome) for the other half, so that we can understand the
relation between a group's actual preferences and hypothetical (stated)
preferences. Connections and lessons from fairness in machine learning are
explored.
