  Motivated by scenarios where data is used for diverse prediction tasks, we
study whether fair representation can be used to guarantee fairness for unknown
tasks and for multiple fairness notions simultaneously. We consider seven group
fairness notions that cover the concepts of independence, separation, and
calibration. Against the backdrop of the fairness impossibility results, we
explore approximate fairness. We prove that, although fair representation might
not guarantee fairness for all prediction tasks, it does guarantee fairness for
an important subset of tasks -- the tasks for which the representation is
discriminative. Specifically, all seven group fairness notions are linearly
controlled by fairness and discriminativeness of the representation. When an
incompatibility exists between different fairness notions, fair and
discriminative representation hits the sweet spot that approximately satisfies
all notions. Motivated by our theoretical findings, we propose to learn both
fair and discriminative representations using pretext loss which
self-supervises learning, and Maximum Mean Discrepancy as a fair regularizer.
Experiments on tabular, image, and face datasets show that using the learned
representation, downstream predictions that we are unaware of when learning the
representation indeed become fairer for seven group fairness notions, and the
fairness guarantees computed from our theoretical results are all valid.
