  In natural language processing (NLP), state-of-the-art (SOTA) semi-supervised
learning (SSL) frameworks have shown great performance on deep pre-trained
language models such as BERT, and are expected to significantly reduce the
demand for manual labeling. However, our empirical studies indicate that these
frameworks are not suitable for lightweight models such as TextCNN, LSTM and
etc. In this work, we develop a new SSL framework called FLiText, which stands
for Faster and Lighter semi-supervised Text classification. FLiText introduces
an inspirer network together with the consistency regularization framework,
which leverages a generalized regular constraint on the lightweight models for
efficient SSL. As a result, FLiText obtains new SOTA performance for
lightweight models across multiple SSL benchmarks on text classification.
Compared with existing SOTA SSL methods on TextCNN, FLiText improves the
accuracy of lightweight model TextCNN from 51.00% to 90.49% on IMDb, 39.8% to
58.06% on Yelp-5, and from 55.3% to 65.08% on Yahoo. In addition, compared with
the fully supervised method on the full dataset, FLiText just uses less than 1%
of labeled data to improve the accuracy by 6.59%, 3.94%, and 3.22% on the
datasets of IMDb, Yelp-5, and Yahoo respectively.
