  With the increasing popularity of conversational search, how to evaluate the
performance of conversational search systems has become an important question
in the IR community. Existing works on conversational search evaluation can
mainly be categorized into two streams: (1) constructing metrics based on
semantic similarity (e.g. BLUE, METEOR and BERTScore), or (2) directly
evaluating the response ranking performance of the system using traditional
search methods (e.g. nDCG, RBP and nERR). However, these methods either ignore
the information need of the user or ignore the mixed-initiative property of
conversational search. This raises the question of how to accurately model user
satisfaction in conversational search scenarios. Since explicitly asking users
to provide satisfaction feedback is difficult, traditional IR studies often
rely on the Cranfield paradigm (i.e., third-party annotation) and user behavior
modeling to estimate user satisfaction in search. However, the feasibility and
effectiveness of these two approaches have not been fully explored in
conversational search. In this paper, we dive into the evaluation of
conversational search from the perspective of user satisfaction. We build a
novel conversational search experimental platform and construct a Chinese
open-domain conversational search behavior dataset containing rich annotations
and search behavior data. We also collect third-party satisfaction annotation
at the session-level and turn-level, to investigate the feasibility of the
Cranfield paradigm in the conversational search scenario. Experimental results
show both some consistency and considerable differences between the user
satisfaction annotations and third-party annotations. We also propose dialog
continuation or ending behavior models (DCEBM) to capture session-level user
satisfaction based on turn-level information.
