  Chatbots, the common moniker for collaborative assistants, are Artificial
Intelligence (AI) software that enables people to naturally interact with them
to get tasks done. Although chatbots have been studied since the dawn of AI,
they have particularly caught the imagination of the public and businesses
since the launch of easy-to-use and general-purpose Large Language Model-based
chatbots like ChatGPT. As businesses look towards chatbots as a potential
technology to engage users, who may be end customers, suppliers, or even their
own employees, proper testing of chatbots is important to address and mitigate
issues of trust related to service or product performance, user satisfaction
and long-term unintended consequences for society. This paper reviews current
practices for chatbot testing, identifies gaps as open problems in pursuit of
user trust, and outlines a path forward.
