  When learning from sensitive data, care must be taken to ensure that training
algorithms address privacy concerns. The canonical Private Aggregation of
Teacher Ensembles, or PATE, computes output labels by aggregating the
predictions of a (possibly distributed) collection of teacher models via a
voting mechanism. The mechanism adds noise to attain a differential privacy
guarantee with respect to the teachers' training data. In this work, we observe
that this use of noise, which makes PATE predictions stochastic, enables new
forms of leakage of sensitive information. For a given input, our adversary
exploits this stochasticity to extract high-fidelity histograms of the votes
submitted by the underlying teachers. From these histograms, the adversary can
learn sensitive attributes of the input such as race, gender, or age. Although
this attack does not directly violate the differential privacy guarantee, it
clearly violates privacy norms and expectations, and would not be possible at
all without the noise inserted to obtain differential privacy. In fact,
counter-intuitively, the attack becomes easier as we add more noise to provide
stronger differential privacy. We hope this encourages future work to consider
privacy holistically rather than treat differential privacy as a panacea.
