  We propose an ensemble learning framework with Poisson sub-sampling to
effectively train a collection of teacher models to issue some differential
privacy (DP) guarantee for training data. Through boosting under DP, a student
model derived from the training data suffers little model degradation from the
models trained with no privacy protection. Our proposed solution leverages upon
two mechanisms, namely: (i) a privacy budget amplification via Poisson
sub-sampling to train a target prediction model that requires less noise to
achieve a same level of privacy budget, and (ii) a combination of the
sub-sampling technique and an ensemble teacher-student learning framework that
introduces DP-preserving noise at the output of the teacher models and
transfers DP-preserving properties via noisy labels. Privacy-preserving student
models are then trained with the noisy labels to learn the knowledge with
DP-protection from the teacher model ensemble. Experimental evidences on spoken
command recognition and continuous speech recognition of Mandarin speech show
that our proposed framework greatly outperforms existing DP-preserving
algorithms in both speech processing tasks.
