  To continuously improve quality and reflect changes in data, machine learning
applications have to regularly retrain and update their core models. We show
that a differential analysis of language model snapshots before and after an
update can reveal a surprising amount of detailed information about changes in
the training data. We propose two new metrics---\emph{differential score} and
\emph{differential rank}---for analyzing the leakage due to updates of natural
language models. We perform leakage analysis using these metrics across models
trained on several different datasets using different methods and
configurations. We discuss the privacy implications of our findings, propose
mitigation strategies and evaluate their effect.
