  Weak supervision enables efficient development of training sets by reducing
the need for ground truth labels. However, the techniques that make weak
supervision attractive -- such as integrating any source of signal to estimate
unknown labels -- also entail the danger that the produced pseudolabels are
highly biased. Surprisingly, given everyday use and the potential for increased
bias, weak supervision has not been studied from the point of view of fairness.
We begin such a study, starting with the observation that even when a fair
model can be built from a dataset with access to ground-truth labels, the
corresponding dataset labeled via weak supervision can be arbitrarily unfair.
To address this, we propose and empirically validate a model for source
unfairness in weak supervision, then introduce a simple counterfactual
fairness-based technique that can mitigate these biases. Theoretically, we show
that it is possible for our approach to simultaneously improve both accuracy
and fairness -- in contrast to standard fairness approaches that suffer from
tradeoffs. Empirically, we show that our technique improves accuracy on weak
supervision baselines by as much as 32\% while reducing demographic parity gap
by 82.5\%. A simple extension of our method aimed at maximizing performance
produces state-of-the-art performance in five out of ten datasets in the WRENCH
benchmark.
