  Conditional Generative Models are now acknowledged an essential tool in
Machine Learning. This paper focuses on their control. While many approaches
aim at disentangling the data through the coordinate-wise control of their
latent representations, another direction is explored in this paper. The
proposed CompVAE handles data with a natural multi-ensemblist structure (i.e.
that can naturally be decomposed into elements). Derived from Bayesian
variational principles, CompVAE learns a latent representation leveraging both
observational and symbolic information. A first contribution of the approach is
that this latent representation supports a compositional generative model,
amenable to multi-ensemblist operations (addition or subtraction of elements in
the composition). This compositional ability is enabled by the invariance and
generality of the whole framework w.r.t. respectively, the order and number of
the elements. The second contribution of the paper is a proof of concept on
synthetic 1D and 2D problems, demonstrating the efficiency of the proposed
approach.
