  Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the
major semi-supervised approaches to improve natural language understanding
(NLU) tasks with massive amount of unlabeled data. However, it's unclear
whether they learn similar representations or they can be effectively combined.
In this paper, we show that TAPT and ST can be complementary with simple TFS
protocol by following TAPT -> Finetuning -> Self-training (TFS) process.
Experimental results show that TFS protocol can effectively utilize unlabeled
data to achieve strong combined gains consistently across six datasets covering
sentiment classification, paraphrase identification, natural language
inference, named entity recognition and dialogue slot classification. We
investigate various semi-supervised settings and consistently show that gains
from TAPT and ST can be strongly additive by following TFS procedure. We hope
that TFS could serve as an important semi-supervised baseline for future NLP
studies.
