  Preference elicitation frameworks feature heavily in the research on
participatory ethical AI tools and provide a viable mechanism to enquire and
incorporate the moral values of various stakeholders. As part of the
elicitation process, surveys about moral preferences, opinions, and judgments
are typically administered only once to each participant. This methodological
practice is reasonable if participants' responses are stable over time such
that, all other relevant factors being held constant, their responses today
will be the same as their responses to the same questions at a later time.
However, we do not know how often that is the case. It is possible that
participants' true moral preferences change, are subject to temporary moods or
whims, or are influenced by environmental factors we don't track. If
participants' moral responses are unstable in such ways, it would raise
important methodological and theoretical issues for how participants' true
moral preferences, opinions, and judgments can be ascertained. We address this
possibility here by asking the same survey participants the same moral
questions about which patient should receive a kidney when only one is
available ten times in ten different sessions over two weeks, varying only
presentation order across sessions. We measured how often participants gave
different responses to simple (Study One) and more complicated (Study Two)
repeated scenarios. On average, the fraction of times participants changed
their responses to controversial scenarios was around 10-18% across studies,
and this instability is observed to have positive associations with response
time and decision-making difficulty. We discuss the implications of these
results for the efficacy of moral preference elicitation, highlighting the role
of response instability in causing value misalignment between stakeholders and
AI tools trained on their moral judgments.
