  We introduce BotSIM, a modular, open-source Bot SIMulation environment with
dialog generation, user simulation and conversation analytics capabilities.
BotSIM aims to serve as a one-stop solution for large-scale data-efficient
end-to-end evaluation, diagnosis and remediation of commercial task-oriented
dialog (TOD) systems to significantly accelerate commercial bot development and
evaluation, reduce cost and time-to-market. BotSIM adopts a layered design
comprising the infrastructure layer, the adaptor layer and the application
layer. The infrastructure layer hosts key models and components to support
BotSIM's major functionalities via a streamlined
"generation-simulation-remediation" pipeline. The adaptor layer is used to
extend BotSIM to accommodate new bot platforms. The application layer provides
a suite of command line tools and a Web App to significantly lower the entry
barrier for BotSIM users such as bot admins or practitioners. In this report,
we focus on the technical designs of various system components. A detailed case
study using Einstein BotBuilder is also presented to show how to apply BotSIM
pipeline for bot evaluation and remediation. The detailed system descriptions
can be found in our system demo paper. The toolkit is available at:
https://github.com/salesforce/BotSIM .
