  Natural Language Understanding (NLU) models can be trained on sensitive
information such as phone numbers, zip-codes etc. Recent literature has focused
on Model Inversion Attacks (ModIvA) that can extract training data from model
parameters. In this work, we present a version of such an attack by extracting
canaries inserted in NLU training data. In the attack, an adversary with
open-box access to the model reconstructs the canaries contained in the model's
training set. We evaluate our approach by performing text completion on
canaries and demonstrate that by using the prefix (non-sensitive) tokens of the
canary, we can generate the full canary. As an example, our attack is able to
reconstruct a four digit code in the training dataset of the NLU model with a
probability of 0.5 in its best configuration. As countermeasures, we identify
several defense mechanisms that, when combined, effectively eliminate the risk
of ModIvA in our experiments.
