  Conversational systems or chatbots are an example of AI-Infused Applications
(AIIA). Chatbots are especially important as they are often the first
interaction of clients with a business and are the entry point of a business
into the AI (Artificial Intelligence) world. The quality of the chatbot is,
therefore, key. However, as is the case in general with AIIAs, it is especially
challenging to assess and control the quality of chatbot systems. Beyond the
inherent statistical nature of these systems, where occasional failure is
acceptable, we identify two major challenges. The first is to release an
initial system that is of sufficient quality such that humans will interact
with it. The second is to maintain the quality, enhance its capabilities,
improve it and make necessary adjustments based on changing user requests or
drift. These challenges exist because it is impossible to predict the real
distribution of user requests and the natural language they will use to express
these requests. Moreover, any empirical distribution of requests is likely to
change over time. This may be due to periodicity, changing usage, and drift of
topics.
  We provide a methodology and set of technologies to address these challenges
and to provide automated assistance through a human-in-the-loop approach. We
notice that it is crucial to connect between the different phases in the
lifecycle of the chatbot development and to make sure it provides its expected
business value. For example, that it frees human agents to deal with tasks
other than answering human users. Our methodology and technologies apply during
chatbot training in the pre-production phase, through to chatbot usage in the
field in the post-production phase. They implement the `test first' paradigm by
assisting in agile design, and support continuous integration through
actionable insights.
