  The proliferation of voice agents in consumer devices requires new tools for
evaluating these systems beyond their technical functionality. This paper
presents a toolkit for the evaluation of Voice User Interfaces (VUIs) with the
intention of measuring the crucial factors of subjective enjoyment in the user
experience. The PUEVA toolkit was constructed using a meta-analysis of existing
literature, structured N=20 and semi-structured N=18 interviews and a within
subjects lab study. The resulting questionnaire contains 35 items that
represent 12 scales in three categories: (1) Personality (2) Usablity and (3)
Enjoyability. The PUEVA Toolkit moves us towards the capacity to evaluate and
compare subjective, joyful experiences in between-subject as well as
within-subject research designs.
