  Task-Oriented Dialogue (TOD) systems are drawing more and more attention in
recent studies. Current methods focus on constructing pre-trained models or
fine-tuning strategies while the evaluation of TOD is limited by a policy
mismatch problem. That is, during evaluation, the user utterances are from the
annotated dataset while these utterances should interact with previous
responses which can have many alternatives besides annotated texts. Therefore,
in this work, we propose an interactive evaluation framework for TOD. We first
build a goal-oriented user simulator based on pre-trained models and then use
the user simulator to interact with the dialogue system to generate dialogues.
Besides, we introduce a sentence-level and a session-level score to measure the
sentence fluency and session coherence in the interactive evaluation.
Experimental results show that RL-based TOD systems trained by our proposed
user simulator can achieve nearly 98% inform and success rates in the
interactive evaluation of MultiWOZ dataset and the proposed scores measure the
response quality besides the inform and success rates. We are hoping that our
work will encourage simulator-based interactive evaluations in the TOD task.
