  Most existing works on disentangled representation learning are solely built
upon an marginal independence assumption: all factors in disentangled
representations should be statistically independent. This assumption is
necessary but definitely not sufficient for the disentangled representations
without additional inductive biases in the modeling process, which is shown
theoretically in recent studies. We argue in this work that disentangled
representations should be characterized by their relation with observable data.
In particular, we formulate such a relation through the concept of mutual
information: the mutual information between each factor of the disentangled
representations and data should be invariant conditioned on values of the other
factors. Together with the widely accepted independence assumption, we further
bridge it with the conditional independence of factors in representations
conditioned on data. Moreover, we note that conditional independence of latent
variables has been imposed on most VAE-type models and InfoGAN due to the
artificial choice of factorized approximate posterior $q(\rvz|\rvx)$ in the
encoders. Such an arrangement of encoders introduces a crucial inductive bias
for disentangled representations. To demonstrate the importance of our proposed
assumption and the related inductive bias, we show in experiments that
violating the assumption leads to decline of disentanglement among factors in
the learned representations.
