  We aim to learn language models for Creole languages for which large volumes
of data are not readily available, and therefore explore the potential transfer
from ancestor languages (the 'Ancestry Transfer Hypothesis'). We find that
standard transfer methods do not facilitate ancestry transfer. Surprisingly,
different from other non-Creole languages, a very distinct two-phase pattern
emerges for Creoles: As our training losses plateau, and language models begin
to overfit on their source languages, perplexity on the Creoles drop. We
explore if this compression phase can lead to practically useful language
models (the 'Ancestry Bottleneck Hypothesis'), but also falsify this. Moreover,
we show that Creoles even exhibit this two-phase pattern even when training on
random, unrelated languages. Thus Creoles seem to be typological outliers and
we speculate whether there is a link between the two observations.
