  Recently, fairness-aware learning have become increasingly crucial, but most
of those methods operate by assuming the availability of fully annotated
demographic group labels. We emphasize that such assumption is unrealistic for
real-world applications since group label annotations are expensive and can
conflict with privacy issues. In this paper, we consider a more practical
scenario, dubbed as Algorithmic Group Fairness with the Partially annotated
Group labels (Fair-PG). We observe that the existing methods to achieve group
fairness perform even worse than the vanilla training, which simply uses full
data only with target labels, under Fair-PG. To address this problem, we
propose a simple Confidence-based Group Label assignment (CGL) strategy that is
readily applicable to any fairness-aware learning method. CGL utilizes an
auxiliary group classifier to assign pseudo group labels, where random labels
are assigned to low confident samples. We first theoretically show that our
method design is better than the vanilla pseudo-labeling strategy in terms of
fairness criteria. Then, we empirically show on several benchmark datasets that
by combining CGL and the state-of-the-art fairness-aware in-processing methods,
the target accuracies and the fairness metrics can be jointly improved compared
to the baselines. Furthermore, we convincingly show that CGL enables to
naturally augment the given group-labeled dataset with external target
label-only datasets so that both accuracy and fairness can be improved. Code is
available at https://github.com/naver-ai/cgl_fairness.
