  Development of machine learning (ML) workflows is a tedious process of
iterative experimentation: developers repeatedly make changes to workflows
until the desired accuracy is attained. We describe our vision for a
"human-in-the-loop" ML system that accelerates this process: by intelligently
tracking changes and intermediate results over time, such a system can enable
rapid iteration, quick responsive feedback, introspection and debugging, and
background execution and automation. We finally describe Helix, our preliminary
attempt at such a system that has already led to speedups of up to 10x on
typical iterative workflows against competing systems.
