  Disentangling data into interpretable and independent factors is critical for
controllable generation tasks. With the availability of labeled data,
supervision can help enforce the separation of specific factors as expected.
However, it is often expensive or even impossible to label every single factor
to achieve fully-supervised disentanglement. In this paper, we adopt a general
setting where all factors that are hard to label or identify are encapsulated
as a single unknown factor. Under this setting, we propose a flexible
weakly-supervised multi-factor disentanglement framework DisUnknown, which
Distills Unknown factors for enabling multi-conditional generation regarding
both labeled and unknown factors. Specifically, a two-stage training approach
is adopted to first disentangle the unknown factor with an effective and robust
training method, and then train the final generator with the proper
disentanglement of all labeled factors utilizing the unknown distillation. To
demonstrate the generalization capacity and scalability of our method, we
evaluate it on multiple benchmark datasets qualitatively and quantitatively and
further apply it to various real-world applications on complicated datasets.
