  Deep neural networks with their large number of parameters are highly
flexible learning systems. The high flexibility in such networks brings with
some serious problems such as overfitting, and regularization is used to
address this problem. A currently popular and effective regularization
technique for controlling the overfitting is dropout. Often, large data
collections required for neural networks contain sensitive information such as
the medical histories of patients, and the privacy of the training data should
be protected. In this paper, we modify the recently proposed variational
dropout technique which provided an elegant Bayesian interpretation to dropout,
and show that the intrinsic noise in the variational dropout can be exploited
to obtain a degree of differential privacy. The iterative nature of training
neural networks presents a challenge for privacy-preserving estimation since
multiple iterations increase the amount of noise added. We overcome this by
using a relaxed notion of differential privacy, called concentrated
differential privacy, which provides tighter estimates on the overall privacy
loss. We demonstrate the accuracy of our privacy-preserving variational dropout
algorithm on benchmark datasets.
