  Human evaluation has been widely accepted as the standard for evaluating
chat-oriented dialogue systems. However, there is a significant variation in
previous work regarding who gets recruited as evaluators. Evaluator groups such
as domain experts, university students, and professional annotators have been
used to assess and compare dialogue systems, although it is unclear to what
extent the choice of an evaluator group can affect results. This paper analyzes
the evaluator group impact on dialogue system evaluation by testing 4
state-of-the-art dialogue systems using 4 distinct evaluator groups. Our
analysis reveals a robustness towards evaluator groups for Likert evaluations
that is not seen for Pairwise, with only minor differences observed when
changing evaluator groups. Furthermore, two notable limitations to this
robustness are observed, which reveal discrepancies between evaluators with
different levels of chatbot expertise and indicate that evaluator objectivity
is beneficial for certain dialogue metrics.
