  A critical bottleneck in supervised machine learning is the need for large
amounts of labeled data which is expensive and time consuming to obtain.
However, it has been shown that a small amount of labeled data, while
insufficient to re-train a model, can be effectively used to generate
human-interpretable labeling functions (LFs). These LFs, in turn, have been
used to generate a large amount of additional noisy labeled data, in a paradigm
that is now commonly referred to as data programming. However, previous
approaches to automatically generate LFs make no attempt to further use the
given labeled data for model training, thus giving up opportunities for
improved performance. Moreover, since the LFs are generated from a relatively
small labeled dataset, they are prone to being noisy, and naively aggregating
these LFs can lead to very poor performance in practice. In this work, we
propose an LF based reweighting framework \ouralgo{} to solve these two
critical limitations. Our algorithm learns a joint model on the (same) labeled
dataset used for LF induction along with any unlabeled data in a
semi-supervised manner, and more critically, reweighs each LF according to its
goodness, influencing its contribution to the semi-supervised loss using a
robust bi-level optimization algorithm. We show that our algorithm
significantly outperforms prior approaches on several text classification
datasets.
