  The fast spreading adoption of machine learning (ML) by companies across
industries poses significant regulatory challenges. One such challenge is
scalability: how can regulatory bodies efficiently audit these ML models,
ensuring that they are fair? In this paper, we initiate the study of
query-based auditing algorithms that can estimate the demographic parity of ML
models in a query-efficient manner. We propose an optimal deterministic
algorithm, as well as a practical randomized, oracle-efficient algorithm with
comparable guarantees. Furthermore, we make inroads into understanding the
optimal query complexity of randomized active fairness estimation algorithms.
Our first exploration of active fairness estimation aims to put AI governance
on firmer theoretical foundations.
