  Assessing the diversity of a dataset of information associated with people is
crucial before using such data for downstream applications. For a given
dataset, this often involves computing the imbalance or disparity in the
empirical marginal distribution of a protected attribute (e.g. gender, dialect,
etc.). However, real-world datasets, such as images from Google Search or
collections of Twitter posts, often do not have protected attributes labeled.
Consequently, to derive disparity measures for such datasets, the elements need
to hand-labeled or crowd-annotated, which are expensive processes.
  We propose a cost-effective approach to approximate the disparity of a given
unlabeled dataset, with respect to a protected attribute, using a control set
of labeled representative examples. Our proposed algorithm uses the pairwise
similarity between elements in the dataset and elements in the control set to
effectively bootstrap an approximation to the disparity of the dataset.
Importantly, we show that using a control set whose size is much smaller than
the size of the dataset is sufficient to achieve a small approximation error.
Further, based on our theoretical framework, we also provide an algorithm to
construct adaptive control sets that achieve smaller approximation errors than
randomly chosen control sets. Simulations on two image datasets and one Twitter
dataset demonstrate the efficacy of our approach (using random and adaptive
control sets) in auditing the diversity of a wide variety of datasets.
