  In recent years, a wide variety of automated machine learning (AutoML)
methods have been proposed to search and generate end-to-end learning
pipelines. While these techniques facilitate the creation of models for
real-world applications, given their black-box nature, the complexity of the
underlying algorithms, and the large number of pipelines they derive, it is
difficult for their developers to debug these systems. It is also challenging
for machine learning experts to select an AutoML system that is well suited for
a given problem or class of problems. In this paper, we present the
PipelineProfiler, an interactive visualization tool that allows the exploration
and comparison of the solution space of machine learning (ML) pipelines
produced by AutoML systems. PipelineProfiler is integrated with Jupyter
Notebook and can be used together with common data science tools to enable a
rich set of analyses of the ML pipelines and provide insights about the
algorithms that generated them. We demonstrate the utility of our tool through
several use cases where PipelineProfiler is used to better understand and
improve a real-world AutoML system. Furthermore, we validate our approach by
presenting a detailed analysis of a think-aloud experiment with six data
scientists who develop and evaluate AutoML tools.
