  From 2016 to 2018, we developed and deployed Chorus, a system that blends
real-time human computation with artificial intelligence (AI) and has
real-world, open conversations with users. We took a top-down approach that
started with a working crowd-powered system, Chorus, and then created a
framework, Evorus, that enables Chorus to automate itself over time. Over our
two-year deployment, more than 420 users talked with Chorus, having over 2,200
conversation sessions. This line of work demonstrated how a crowd-powered
conversational assistant can be automated over time, and more importantly, how
such a system can be deployed to talk with real users to help them with their
everyday tasks. This position paper discusses two sets of challenges that we
explored during the development and deployment of Chorus and Evorus: the
challenges that come from being an "agent" and those that arise from the subset
of conversations that are more difficult to automate.
