  Chatbots are envisioned to dramatically change the future of Software
Engineering, allowing practitioners to chat and inquire about their software
projects and interact with different services using natural language. At the
heart of every chatbot is a Natural Language Understanding (NLU) component that
enables the chatbot to understand natural language input. Recently, many NLU
platforms were provided to serve as an off-the-shelf NLU component for
chatbots, however, selecting the best NLU for Software Engineering chatbots
remains an open challenge.
  Therefore, in this paper, we evaluate four of the most commonly used NLUs,
namely IBM Watson, Google Dialogflow, Rasa, and Microsoft LUIS to shed light on
which NLU should be used in Software Engineering based chatbots. Specifically,
we examine the NLUs' performance in classifying intents, confidence scores
stability, and extracting entities. To evaluate the NLUs, we use two datasets
that reflect two common tasks performed by Software Engineering practitioners,
1) the task of chatting with the chatbot to ask questions about software
repositories 2) the task of asking development questions on Q&A forums (e.g.,
Stack Overflow). According to our findings, IBM Watson is the best performing
NLU when considering the three aspects (intents classification, confidence
scores, and entity extraction). However, the results from each individual
aspect show that, in intents classification, IBM Watson performs the best with
an F1-measure > 84%, but in confidence scores, Rasa comes on top with a median
confidence score higher than 0.91. Our results also show that all NLUs, except
for Dialogflow, generally provide trustable confidence scores. For entity
extraction, Microsoft LUIS and IBM Watson outperform other NLUs in the two SE
tasks. Our results provide guidance to software engineering practitioners when
deciding which NLU to use in their chatbots.
