  Human-in-the-loop aims to train an accurate prediction model with minimum
cost by integrating human knowledge and experience. Humans can provide training
data for machine learning applications and directly accomplish tasks that are
hard for computers in the pipeline with the help of machine-based approaches.
In this paper, we survey existing works on human-in-the-loop from a data
perspective and classify them into three categories with a progressive
relationship: (1) the work of improving model performance from data processing,
(2) the work of improving model performance through interventional model
training, and (3) the design of the system independent human-in-the-loop. Using
the above categorization, we summarize major approaches in the field; along
with their technical strengths/ weaknesses, we have simple classification and
discussion in natural language processing, computer vision, and others.
Besides, we provide some open challenges and opportunities. This survey intends
to provide a high-level summarization for human-in-the-loop and motivates
interested readers to consider approaches for designing effective
human-in-the-loop solutions.
