  This paper explores the use of latent bootstrapping, an alternative
self-supervision technique, for pretraining language models. Unlike the typical
practice of using self-supervision on discrete subwords, latent bootstrapping
leverages contextualized embeddings for a richer supervision signal. We conduct
experiments to assess how effective this approach is for acquiring linguistic
knowledge from limited resources. Specifically, our experiments are based on
the BabyLM shared task, which includes pretraining on two small curated corpora
and an evaluation on four linguistic benchmarks.
