  This thesis focuses on improving the pre-training of natural language models
using unsupervised raw data to make them more efficient and aligned with
downstream applications.
  In the first part, we introduce three alternative pre-training objectives to
BERT's Masked Language Modeling (MLM), namely Random Token Substitution (RTS),
Cluster-based Random Token Substitution (C-RTS), and Swapped Language Modeling
(SLM). These objectives involve token swapping instead of masking, with RTS and
C-RTS aiming to predict token originality and SLM predicting the original token
values. Results show that RTS and C-RTS require less pre-training time while
maintaining performance comparable to MLM. Surprisingly, SLM outperforms MLM on
certain tasks despite using the same computational budget.
  In the second part, we proposes self-supervised pre-training tasks that align
structurally with downstream applications, reducing the need for labeled data.
We use large corpora like Wikipedia and CC-News to train models to recognize if
text spans originate from the same paragraph or document in several ways. By
doing continuous pre-training, starting from existing models like RoBERTa,
ELECTRA, DeBERTa, BART, and T5, we demonstrate significant performance
improvements in tasks like Fact Verification, Answer Sentence Selection, and
Summarization. These improvements are especially pronounced when limited
annotation data is available. The proposed objectives also achieve
state-of-the-art results on various benchmark datasets, including FEVER (dev
set), ASNQ, WikiQA, and TREC-QA, as well as enhancing the quality of summaries.
Importantly, these techniques can be easily integrated with other methods
without altering the internal structure of Transformer models, making them
versatile for various NLP applications.
