  In many applications, training machine learning models involves using large
amounts of human-annotated data. Obtaining precise labels for the data is
expensive. Instead, training with weak supervision provides a low-cost
alternative. We propose a novel weak supervision algorithm that processes noisy
labels, i.e., weak signals, while also considering features of the training
data to produce accurate labels for training. Our method searches over
classifiers of the data representation to find plausible labelings. We call
this paradigm data consistent weak supervision. A key facet of our framework is
that we are able to estimate labels for data examples low or no coverage from
the weak supervision. In addition, we make no assumptions about the joint
distribution of the weak signals and true labels of the data. Instead, we use
weak signals and the data features to solve a constrained optimization that
enforces data consistency among the labels we generate. Empirical evaluation of
our method on different datasets shows that it significantly outperforms
state-of-the-art weak supervision methods on both text and image classification
tasks.
