  Applying differential privacy (DP) by means of the DP-SGD algorithm to
protect individual data points during training is becoming increasingly popular
in NLP. However, the choice of granularity at which DP is applied is often
neglected. For example, neural machine translation (NMT) typically operates on
the sentence-level granularity. From the perspective of DP, this setup assumes
that each sentence belongs to a single person and any two sentences in the
training dataset are independent. This assumption is however violated in many
real-world NMT datasets, e.g. those including dialogues. For proper application
of DP we thus must shift from sentences to entire documents. In this paper, we
investigate NMT at both the sentence and document levels, analyzing the
privacy/utility trade-off for both scenarios, and evaluating the risks of not
using the appropriate privacy granularity in terms of leaking personally
identifiable information (PII). Our findings indicate that the document-level
NMT system is more resistant to membership inference attacks, emphasizing the
significance of using the appropriate granularity when working with DP.
