  Metric Elicitation (ME) is a framework for eliciting classification metrics
that better align with implicit user preferences based on the task and context.
The existing ME strategy so far is based on the assumption that users can most
easily provide preference feedback over classifier statistics such as confusion
matrices. This work examines ME, by providing a first ever implementation of
the ME strategy. Specifically, we create a web-based ME interface and conduct a
user study that elicits users' preferred metrics in a binary classification
setting. We discuss the study findings and present guidelines for future
research in this direction.
