  Synthetic data is often presented as a method for sharing sensitive
information in a privacy-preserving manner by reproducing the global
statistical properties of the original data without disclosing sensitive
information about any individual. In practice, as with other anonymization
methods, privacy risks cannot be entirely eliminated. The residual privacy
risks need instead to be ex-post assessed. We present Anonymeter, a statistical
framework to jointly quantify different types of privacy risks in synthetic
tabular datasets. We equip this framework with attack-based evaluations for the
singling out, linkability, and inference risks, the three key indicators of
factual anonymization according to the European General Data Protection
Regulation (GDPR). To the best of our knowledge, we are the first to introduce
a coherent and legally aligned evaluation of these three privacy risks for
synthetic data, and to design privacy attacks which model directly the singling
out and linkability risks. We demonstrate the effectiveness of our methods by
conducting an extensive set of experiments that measure the privacy risks of
data with deliberately inserted privacy leakages, and of synthetic data
generated with and without differential privacy. Our results highlight that the
three privacy risks reported by our framework scale linearly with the amount of
privacy leakage in the data. Furthermore, we observe that synthetic data
exhibits the lowest vulnerability against linkability, indicating one-to-one
relationships between real and synthetic data records are not preserved.
Finally, we demonstrate quantitatively that Anonymeter outperforms existing
synthetic data privacy evaluation frameworks both in terms of detecting privacy
leaks, as well as computation speed. To contribute to a privacy-conscious usage
of synthetic data, we open source Anonymeter at
https://github.com/statice/anonymeter.
