  We propose a new family of label randomizers for training regression models
under the constraint of label differential privacy (DP). In particular, we
leverage the trade-offs between bias and variance to construct better label
randomizers depending on a privately estimated prior distribution over the
labels. We demonstrate that these randomizers achieve state-of-the-art
privacy-utility trade-offs on several datasets, highlighting the importance of
reducing bias when training neural networks with label DP. We also provide
theoretical results shedding light on the structural properties of the optimal
unbiased randomizers.
