  We study the problem of collecting a cohort or set that is balanced with
respect to sensitive groups when group membership is unavailable or prohibited
from use at deployment time. Specifically, our deployment-time collection
mechanism does not reveal significantly more about the group membership of any
individual sample than can be ascertained from base rates alone. To do this, we
study a learner that can use a small set of labeled data to train a proxy
function that can later be used for this filtering or selection task. We then
associate the range of the proxy function with sampling probabilities; given a
new example, we classify it using our proxy function and then select it with
probability corresponding to its proxy classification. Importantly, we require
that the proxy classification does not reveal significantly more information
about the sensitive group membership of any individual example compared to
population base rates alone (i.e., the level of disclosure should be
controlled) and show that we can find such a proxy in a sample- and
oracle-efficient manner. Finally, we experimentally evaluate our algorithm and
analyze its generalization properties.
