  Evaluating and understanding the inappropriateness of chatbot behaviors can
be challenging, particularly for chatbot designers without technical
backgrounds. To democratize the debugging process of chatbot misbehaviors for
non-technical designers, we propose a framework that leverages dialogue act
(DA) modeling to automate the evaluation and explanation of chatbot response
inappropriateness. The framework first produces characterizations of
context-aware DAs based on discourse analysis theory and real-world
human-chatbot transcripts. It then automatically extracts features to identify
the appropriateness level of a response and can explain the causes of the
inappropriate response by examining the DA mismatch between the response and
its conversational context. Using interview chatbots as a testbed, our
framework achieves comparable classification accuracy with higher
explainability and fewer computational resources than the deep learning
baseline, making it the first step in utilizing DAs for chatbot response
appropriateness evaluation and explanation.
