  The scalability of the labeling process and the attainable quality of labels
have become limiting factors for many applications of machine learning. The
programmatic creation of labeled datasets via the synthesis of noisy heuristics
provides a promising avenue to address this problem. We propose to improve
modeling of latent class variables in the programmatic creation of labeled
datasets by incorporating pairwise feedback into the process. We discuss the
ease with which such pairwise feedback can be obtained or generated in many
application domains. Our experiments show that even a small number of sources
of pairwise feedback can substantially improve the quality of the posterior
estimate of the latent class variable.
