  Deep neural networks achieve remarkable performances on a wide range of tasks
with the aid of large-scale labeled datasets. Yet these datasets are
time-consuming and labor-exhaustive to obtain on realistic tasks. To mitigate
the requirement for labeled data, self-training is widely used in
semi-supervised learning by iteratively assigning pseudo labels to unlabeled
samples. Despite its popularity, self-training is well-believed to be
unreliable and often leads to training instability. Our experimental studies
further reveal that the bias in semi-supervised learning arises from both the
problem itself and the inappropriate training with potentially incorrect pseudo
labels, which accumulates the error in the iterative self-training process. To
reduce the above bias, we propose Debiased Self-Training (DST). First, the
generation and utilization of pseudo labels are decoupled by two
parameter-independent classifier heads to avoid direct error accumulation.
Second, we estimate the worst case of self-training bias, where the pseudo
labeling function is accurate on labeled samples, yet makes as many mistakes as
possible on unlabeled samples. We then adversarially optimize the
representations to improve the quality of pseudo labels by avoiding the worst
case. Extensive experiments justify that DST achieves an average improvement of
6.3% against state-of-the-art methods on standard semi-supervised learning
benchmark datasets and 18.9%$ against FixMatch on 13 diverse tasks.
Furthermore, DST can be seamlessly adapted to other self-training methods and
help stabilize their training and balance performance across classes in both
cases of training from scratch and finetuning from pre-trained models.
