  User simulators are essential for training reinforcement learning (RL) based
dialog models. The performance of the simulator directly impacts the RL policy.
However, building a good user simulator that models real user behaviors is
challenging. We propose a method of standardizing user simulator building that
can be used by the community to compare dialog system quality using the same
set of user simulators fairly. We present implementations of six user
simulators trained with different dialog planning and generation methods. We
then calculate a set of automatic metrics to evaluate the quality of these
simulators both directly and indirectly. We also ask human users to assess the
simulators directly and indirectly by rating the simulated dialogs and
interacting with the trained systems. This paper presents a comprehensive
evaluation framework for user simulator study and provides a better
understanding of the pros and cons of different user simulators, as well as
their impacts on the trained systems.
