  We make two theoretical contributions to disentanglement learning by (a)
defining precise semantics of disentangled representations, and (b)
establishing robust metrics for evaluation. First, we characterize the concept
"disentangled representations" used in supervised and unsupervised methods
along three dimensions-informativeness, separability and interpretability -
which can be expressed and quantified explicitly using information-theoretic
constructs. This helps explain the behaviors of several well-known
disentanglement learning models. We then propose robust metrics for measuring
informativeness, separability and interpretability. Through a comprehensive
suite of experiments, we show that our metrics correctly characterize the
representations learned by different methods and are consistent with
qualitative (visual) results. Thus, the metrics allow disentanglement learning
methods to be compared on a fair ground. We also empirically uncovered new
interesting properties of VAE-based methods and interpreted them with our
formulation. These findings are promising and hopefully will encourage the
design of more theoretically driven models for learning disentangled
representations.
