  Evaluation of intelligent assistants in large-scale and online settings
remains an open challenge. User behavior-based online evaluation metrics have
demonstrated great effectiveness for monitoring large-scale web search and
recommender systems. Therefore, we consider predicting user engagement status
as the very first and critical step to online evaluation for intelligent
assistants. In this work, we first proposed a novel framework for classifying
user engagement status into four categories -- fulfillment, continuation,
reformulation and abandonment. We then demonstrated how to design simple but
indicative metrics based on the framework to quantify user engagement levels.
We also aim for automating user engagement prediction with machine learning
methods. We compare various models and features for predicting engagement
status using four real-world datasets. We conducted detailed analyses on
features and failure cases to discuss the performance of current models as well
as challenges.
