  New regulations and increased awareness of data privacy have led to the
deployment of new and more efficient differentially private mechanisms across
public institutions and industries. Ensuring the correctness of these
mechanisms is therefore crucial to ensure the proper protection of data.
However, since differential privacy is a property of the mechanism itself, and
not of an individual output, testing whether a mechanism is differentially
private is not a trivial task. While ad hoc testing techniques exist under
specific assumptions, no concerted effort has been made by the research
community to develop a flexible and extendable tool for testing differentially
private mechanisms. This paper introduces DP-Auditorium as a step advancing
research in this direction. DP-Auditorium abstracts the problem of testing
differential privacy into two steps: (1) measuring the distance between
distributions, and (2) finding neighboring datasets where a mechanism generates
output distributions maximizing such distance. From a technical point of view,
we propose three new algorithms for evaluating the distance between
distributions. While these algorithms are well-established in the statistics
community, we provide new estimation guarantees that exploit the fact that we
are only interested in verifying whether a mechanism is differentially private,
and not in obtaining an exact estimate of the distance between two
distributions. DP-Auditorium is easily extensible, as demonstrated in this
paper by implementing a well-known approximate differential privacy testing
algorithm into our library. We provide an extensive comparison to date of
multiple testers across varying sample sizes and differential privacy
parameters, demonstrating that there is no single tester that dominates all
others, and that a combination of different techniques is required to ensure
proper testing of mechanisms.
