  In recent years, the proliferation of smart mobile devices has lead to the
gradual integration of search functionality within mobile platforms. This has
created an incentive to move away from the "ten blue links'' metaphor, as
mobile users are less likely to click on them, expecting to get the answer
directly from the snippets. In turn, this has revived the interest in Question
Answering. Then, along came chatbots, conversational systems, and messaging
platforms, where the user needs could be better served with the system asking
follow-up questions in order to better understand the user's intent. While
typically a user would expect a single response at any utterance, a system
could also return multiple options for the user to select from, based on
different system understandings of the user's intent. However, this possibility
should not be overused, as this practice could confuse and/or annoy the user.
How to produce good variable-length lists, given the conflicting objectives of
staying short while maximizing the likelihood of having a correct answer
included in the list, is an underexplored problem. It is also unclear how to
evaluate a system that tries to do that. Here we aim to bridge this gap. In
particular, we define some necessary and some optional properties that an
evaluation measure fit for this purpose should have. We further show that
existing evaluation measures from the IR tradition are not entirely suitable
for this setup, and we propose novel evaluation measures that address it
satisfactorily.
