  Semantic image synthesis (SIS) refers to the problem of generating realistic
imagery given a semantic segmentation mask that defines the spatial layout of
object classes. Most of the approaches in the literature, other than the
quality of the generated images, put effort in finding solutions to increase
the generation diversity in terms of style i.e. texture. However, they all
neglect a different feature, which is the possibility of manipulating the
layout provided by the mask. Currently, the only way to do so is manually by
means of graphical users interfaces. In this paper, we describe a network
architecture to address the problem of automatically manipulating or generating
the shape of object classes in semantic segmentation masks, with specific focus
on human faces. Our proposed model allows embedding the mask class-wise into a
latent space where each class embedding can be independently edited. Then, a
bi-directional LSTM block and a convolutional decoder output a new, locally
manipulated mask. We report quantitative and qualitative results on the
CelebMask-HQ dataset, which show our model can both faithfully reconstruct and
modify a segmentation mask at the class level. Also, we show our model can be
put before a SIS generator, opening the way to a fully automatic generation
control of both shape and texture. Code available at
https://github.com/TFonta/Semantic-VAE.
