  Applications of Differential Privacy (DP) in NLP must distinguish between the
syntactic level on which a proposed mechanism operates, often taking the form
of $\textit{word-level}$ or $\textit{document-level}$ privatization. Recently,
several word-level $\textit{Metric}$ Differential Privacy approaches have been
proposed, which rely on this generalized DP notion for operating in word
embedding spaces. These approaches, however, often fail to produce semantically
coherent textual outputs, and their application at the sentence- or
document-level is only possible by a basic composition of word perturbations.
In this work, we strive to address these challenges by operating
$\textit{between}$ the word and sentence levels, namely with
$\textit{collocations}$. By perturbing n-grams rather than single words, we
devise a method where composed privatized outputs have higher semantic
coherence and variable length. This is accomplished by constructing an
embedding model based on frequently occurring word groups, in which unigram
words co-exist with bi- and trigram collocations. We evaluate our method in
utility and privacy tests, which make a clear case for tokenization strategies
beyond the word level.
