  We report on an investigation into how different types of failures in a voice
user interface (VUI) affects user frustration. To this end, we conducted a
pilot user study ($n=10$) and a main user study ($n=30$), both with a simple
voice-operated calendar application that we built using the Alexa Skills Kit.
In our pilot study, we identified three major failure types as perceived by the
users, namely, Reason Unknown, Speech Misrecognition, and Utterance Pattern
Match Failure, along with more fine-grained failure types from the developer's
viewpoint such as Intent Pattern Match Failure and Intent Misclassification.
Then, in our main study, we set up three user tasks that were designed to each
induce a specific failure type, and collected user frustration ratings for each
task. Our main findings are:
  (a)Users may be relatively tolerant to user-perceived Speech Misrecognition,
and not so to user-perceived Reason Unknown and Utterance Mattern Match
Failures;
  (b)Regarding the relationship between developer-perceived and user-perceived
failure types, 68.8\% of developer-perceived Intent Misclassification instances
caused user-perceived Reason Unkown failures.
  From (a) and (b), a practical design implication would be to try to prevent
Intent Misclassification from happening by carefully crafting the utterance
patterns for each intent.
