  Graph convolutional networks (GCNs) are a powerful architecture for
representation learning on documents that naturally occur as graphs, e.g.,
citation or social networks. However, sensitive personal information, such as
documents with people's profiles or relationships as edges, are prone to
privacy leaks, as the trained model might reveal the original input. Although
differential privacy (DP) offers a well-founded privacy-preserving framework,
GCNs pose theoretical and practical challenges due to their training specifics.
We address these challenges by adapting differentially-private gradient-based
training to GCNs and conduct experiments using two optimizers on five NLP
datasets in two languages. We propose a simple yet efficient method based on
random graph splits that not only improves the baseline privacy bounds by a
factor of 2.7 while retaining competitive F1 scores, but also provides strong
privacy guarantees of epsilon = 1.0. We show that, under certain modeling
choices, privacy-preserving GCNs perform up to 90% of their non-private
variants, while formally guaranteeing strong privacy measures.
