  Learning discriminative powerful representations is a crucial step for
machine learning systems. Introducing invariance against arbitrary nuisance or
sensitive attributes while performing well on specific tasks is an important
problem in representation learning. This is mostly approached by purging the
sensitive information from learned representations. In this paper, we propose a
novel disentanglement approach to invariant representation problem. We
disentangle the meaningful and sensitive representations by enforcing
orthogonality constraints as a proxy for independence. We explicitly enforce
the meaningful representation to be agnostic to sensitive information by
entropy maximization. The proposed approach is evaluated on five publicly
available datasets and compared with state of the art methods for learning
fairness and invariance achieving the state of the art performance on three
datasets and comparable performance on the rest. Further, we perform an
ablative study to evaluate the effect of each component.
