  Artificial Intelligence (AI) has burrowed into our lives in various aspects;
however, without appropriate testing, deployed AI systems are often being
criticized to fail in critical and embarrassing cases. Existing testing
approaches mainly depend on fixed and pre-defined datasets, providing a limited
testing coverage. In this paper, we propose the concept of proactive testing to
dynamically generate testing data and evaluate the performance of AI systems.
We further introduce Challenge.AI, a new crowd system that features the
integration of crowdsourcing and machine learning techniques in the process of
error generation, error validation, error categorization, and error analysis.
We present experiences and insights into a participatory design with AI
developers. The evaluation shows that the crowd workflow is more effective with
the help of machine learning techniques. AI developers found that our system
can help them discover unknown errors made by the AI models, and engage in the
process of proactive testing.
