  Word meaning change can be inferred from drifts of time-varying word
embeddings. However, temporal data may be too sparse to build robust word
embeddings and to discriminate significant drifts from noise. In this paper, we
compare three models to learn diachronic word embeddings on scarce data:
incremental updating of a Skip-Gram from Kim et al. (2014), dynamic filtering
from Bamler and Mandt (2017), and dynamic Bernoulli embeddings from Rudolph and
Blei (2018). In particular, we study the performance of different
initialisation schemes and emphasise what characteristics of each model are
more suitable to data scarcity, relying on the distribution of detected drifts.
Finally, we regularise the loss of these models to better adapt to scarce data.
