  This review gives an extensive overview of evaluation methods for
task-oriented dialogue systems, paying special attention to practical
applications of dialogue systems, for example for customer service. The review
(1) provides an overview of the used constructs and metrics in previous work,
(2) discusses challenges in the context of dialogue system evaluation and (3)
develops a research agenda for the future of dialogue system evaluation. We
conducted a systematic review of four databases (ACL, ACM, IEEE and Web of
Science), which after screening resulted in 122 studies. Those studies were
carefully analysed for the constructs and methods they proposed for evaluation.
We found a wide variety in both constructs and methods. Especially the
operationalisation is not always clearly reported. Newer developments
concerning large language models are discussed in two contexts: to power
dialogue systems and to use in the evaluation process. We hope that future work
will take a more critical approach to the operationalisation and specification
of the used constructs. To work towards this aim, this review ends with
recommendations for evaluation and suggestions for outstanding questions.
