  Training deep neural networks (DNNs) under weak supervision has attracted
increasing research attention as it can significantly reduce the annotation
cost. However, labels from weak supervision can be noisy, and the high capacity
of DNNs enables them to easily overfit the label noise, resulting in poor
generalization. Recent methods leverage self-training to build noise-resistant
models, in which a teacher trained under weak supervision is used to provide
highly confident labels for teaching the students. Nevertheless, the teacher
derived from such frameworks may have fitted a substantial amount of noise and
therefore produce incorrect pseudo-labels with high confidence, leading to
severe error propagation. In this work, we propose Meta Self-Refinement (MSR),
a noise-resistant learning framework, to effectively combat label noise from
weak supervision. Instead of relying on a fixed teacher trained with noisy
labels, we encourage the teacher to refine its pseudo-labels. At each training
step, MSR performs a meta gradient descent on the current mini-batch to
maximize the student performance on a clean validation set. Extensive
experimentation on eight NLP benchmarks demonstrates that MSR is robust against
label noise in all settings and outperforms state-of-the-art methods by up to
11.4% in accuracy and 9.26% in F1 score.
