  Many circumstances of practical importance have performance or success
metrics which exist implicitly---in the eye of the beholder, so to speak.
Tuning aspects of such problems requires working without defined metrics and
only considering pairwise comparisons or rankings. In this paper, we review an
existing Bayesian optimization strategy for determining most-preferred
outcomes, and identify an adaptation to allow it to handle ties. We then
discuss some of the issues we have encountered when humans use this
optimization strategy to optimize coloring a piece of abstract artwork. We hope
that, by participating in this workshop, we can learn how other researchers
encounter difficulties unique to working with humans in the loop.
