  Tuning the hyperparameters in the differentially private stochastic gradient
descent (DPSGD) is a fundamental challenge. Unlike the typical SGD, private
datasets cannot be used many times for hyperparameter search in DPSGD; e.g.,
via a grid search. Therefore, there is an essential need for algorithms that,
within a given search space, can find near-optimal hyperparameters for the best
achievable privacy-utility tradeoffs efficiently. We formulate this problem
into a general optimization framework for establishing a desirable
privacy-utility tradeoff, and systematically study three cost-effective
algorithms for being used in the proposed framework: evolutionary, Bayesian,
and reinforcement learning. Our experiments, for hyperparameter tuning in DPSGD
conducted on MNIST and CIFAR-10 datasets, show that these three algorithms
significantly outperform the widely used grid search baseline. As this paper
offers a first-of-a-kind framework for hyperparameter tuning in DPSGD, we
discuss existing challenges and open directions for future studies. As we
believe our work has implications to be utilized in the pipeline of private
deep learning, we open-source our code at
https://github.com/AmanPriyanshu/DP-HyperparamTuning.
