  Differential privacy has become the dominant standard in the research
community for strong privacy protection. There has been a flood of research
into query answering algorithms that meet this standard. Algorithms are
becoming increasingly complex, and in particular, the performance of many
emerging algorithms is {\em data dependent}, meaning the distribution of the
noise added to query answers may change depending on the input data.
Theoretical analysis typically only considers the worst case, making empirical
study of average case performance increasingly important.
  In this paper we propose a set of evaluation principles which we argue are
essential for sound evaluation. Based on these principles we propose DPBench, a
novel evaluation framework for standardized evaluation of privacy algorithms.
We then apply our benchmark to evaluate algorithms for answering 1- and
2-dimensional range queries. The result is a thorough empirical study of 15
published algorithms on a total of 27 datasets that offers new insights into
algorithm behavior---in particular the influence of dataset scale and
shape---and a more complete characterization of the state of the art. Our
methodology is able to resolve inconsistencies in prior empirical studies and
place algorithm performance in context through comparison to simple baselines.
Finally, we pose open research questions which we hope will guide future
algorithm design.
