  Binary concepts are empirically used by humans to generalize efficiently. And
they are based on Bernoulli distribution which is the building block of
information. These concepts span both low-level and high-level features such as
"large vs small" and "a neuron is active or inactive". Binary concepts are
ubiquitous features and can be used to transfer knowledge to improve model
generalization. We propose a novel binarized regularization to facilitate
learning of binary concepts to improve the quality of data generation in
autoencoders. We introduce a binarizing hyperparameter $r$ in data generation
process to disentangle the latent space symmetrically. We demonstrate that this
method can be applied easily to existing variational autoencoder (VAE) variants
to encourage symmetric disentanglement, improve reconstruction quality, and
prevent posterior collapse without computation overhead. We also demonstrate
that this method can boost existing models to learn more transferable
representations and generate more representative samples for the input
distribution which can alleviate catastrophic forgetting using generative
replay under continual learning settings.
