  When a model informs decisions about people, distribution shifts can create
undue disparities. However, it is hard for external entities to check for
distribution shift, as the model and its training set are often proprietary. In
this paper, we introduce and study a black-box auditing method to detect cases
of distribution shift that lead to a performance disparity of the model across
demographic groups. By extending techniques used in membership and property
inference attacks -- which are designed to expose private information from
learned models -- we demonstrate that an external auditor can gain the
information needed to identify these distribution shifts solely by querying the
model. Our experimental results on real-world datasets show that this approach
is effective, achieving 80--100% AUC-ROC in detecting shifts involving the
underrepresentation of a demographic group in the training set. Researchers and
investigative journalists can use our tools to perform non-collaborative audits
of proprietary models and expose cases of underrepresentation in the training
datasets.
