  Disentanglement is a highly desirable property of representation owing to its
similarity to human understanding and reasoning. Many works achieve
disentanglement upon information bottlenecks (IB). Despite their elegant
mathematical foundations, the IB branch usually exhibits lower performance. In
order to provide an insight into the problem, we develop an annealing test to
calculate the information freezing point (IFP), which is a transition state to
freeze information into the latent variables. We also explore these clues or
inductive biases for separating the entangled factors according to the
differences in the IFP distributions. We found the existing approaches suffer
from the information diffusion problem, according to which the increased
information diffuses in all latent variables.
  Based on this insight, we propose a novel disentanglement framework, termed
the distilling entangled factor (DEFT), to address the information diffusion
problem by scaling backward information. DEFT applies a multistage training
strategy, including multigroup encoders with different learning rates and
piecewise disentanglement pressure, to disentangle the factors stage by stage.
We evaluate DEFT on three variants of dSprite and SmallNORB, which show
low-variance and high-level disentanglement scores. Furthermore, the experiment
under the correlative factors shows incapable of TC-based approaches. DEFT also
exhibits a competitive performance in the unsupervised setting.
