  Deep learning-based language models have achieved state-of-the-art results in
a number of applications including sentiment analysis, topic labelling, intent
classification and others. Obtaining text representations or embeddings using
these models presents the possibility of encoding personally identifiable
information learned from language and context cues that may present a risk to
reputation or privacy. To ameliorate these issues, we propose Context-Aware
Private Embeddings (CAPE), a novel approach which preserves privacy during
training of embeddings. To maintain the privacy of text representations, CAPE
applies calibrated noise through differential privacy, preserving the encoded
semantic links while obscuring sensitive information. In addition, CAPE employs
an adversarial training regime that obscures identified private variables.
Experimental results demonstrate that the proposed approach reduces private
information leakage better than either single intervention.
