  Preference elicitation leverages AI or optimization to learn stakeholder
preferences in settings ranging from marketing to public policy. The online
robust preference elicitation procedure of arXiv:2003.01899 has been shown in
simulation to outperform various other elicitation procedures in terms of
effectively learning individuals' true utilities. However, as with any
simulation, the method makes a series of assumptions that cannot easily be
verified to hold true beyond simulation. Thus, we propose to validate the
robust method's performance using real users, focusing on the particular
challenge of selecting policies for prioritizing COVID-19 patients for scarce
hospital resources during the pandemic. To this end, we develop an online
platform for preference elicitation where users report their preferences
between alternatives over a moderate number of pairwise comparisons chosen by a
particular elicitation procedure. We recruit 193 Amazon Mechanical Turk (MTurk)
workers to report their preferences and demonstrate that the robust method
outperforms asking random queries by 21%, the next best performing method in
the simulated results of arXiv:2003.01899, in terms of recommending policies
with a higher utility.
