  We propose a novel and simple method for semi-supervised text classification.
The method stems from the hypothesis that a classifier with pretrained word
embeddings always outperforms the same classifier with randomly initialized
word embeddings, as empirically observed in NLP tasks. Our method first builds
two sets of classifiers as a form of model ensemble, and then initializes their
word embeddings differently: one using random, the other using pretrained word
embeddings. We focus on different predictions between the two classifiers on
unlabeled data while following the self-training framework. We also use
early-stopping in meta-epoch to improve the performance of our method. Our
method, Delta-training, outperforms the self-training and the co-training
framework in 4 different text classification datasets, showing robustness
against error accumulation.
