  Digital assistants are experiencing rapid growth due to their ability to
assist users with day-to-day tasks where most dialogues are happening
multi-turn. However, evaluating multi-turn dialogues remains challenging,
especially at scale. We suggest a context-sensitive method to estimate the
turn-level satisfaction for dialogue considering various types of user
preferences. The costs of interactions between users and dialogue systems are
formulated using a budget consumption concept. We assume users have an initial
interaction budget for a dialogue formed based on the task complexity and that
each turn has a cost. When the task is completed, or the budget has been
exhausted, users quit the dialogue. We demonstrate our method's effectiveness
by extensive experimentation with a simulated dialogue platform and real
multi-turn dialogues.
