  An end-to-end data integration system requires human feedback in several
phases, including collecting training data for entity matching, debugging the
resulting clusters, confirming transformations applied on these clusters for
data standardization, and finally, reducing each cluster to a single, canonical
representation (or "golden record"). The traditional wisdom is to sequentially
apply the human feedback, obtained by asking specific questions, within some
budget in each phase. However, these questions are highly correlated; the
answer to one can influence the outcome of any of the phases of the pipeline.
Hence, interleaving them has the potential to offer significant benefits.
  In this paper, we propose a human-in-the-loop framework that interleaves
different types of questions to optimize human involvement. We propose benefit
models to measure the quality improvement from asking a question, and cost
models to measure the human time it takes to answer a question. We develop a
question scheduling framework that judiciously selects questions to maximize
the accuracy of the final golden records. Experimental results on three
real-world datasets show that our holistic method significantly improves the
quality of golden records from 70% to 90%, compared with the state-of-the-art
approaches.
