  Partitioning a set of elements into an unknown number of mutually exclusive
subsets is essential in many machine learning problems. However, assigning
elements, such as samples in a dataset or neurons in a network layer, to an
unknown and discrete number of subsets is inherently non-differentiable,
prohibiting end-to-end gradient-based optimization of parameters. We overcome
this limitation by proposing a novel two-step method for inferring partitions,
which allows its usage in variational inference tasks. This new approach
enables reparameterized gradients with respect to the parameters of the new
random partition model. Our method works by inferring the number of elements
per subset and, second, by filling these subsets in a learned order. We
highlight the versatility of our general-purpose approach on three different
challenging experiments: variational clustering, inference of shared and
independent generative factors under weak supervision, and multitask learning.
