  The injection of syntactic information in Variational AutoEncoders (VAEs) has
been shown to result in an overall improvement of performances and
generalisation. An effective strategy to achieve such a goal is to separate the
encoding of distributional semantic features and syntactic structures into
heterogeneous latent spaces via multi-task learning or dual encoder
architectures. However, existing works employing such techniques are limited to
LSTM-based VAEs. In this paper, we investigate latent space separation methods
for structural syntactic injection in Transformer-based VAE architectures
(i.e., Optimus). Specifically, we explore how syntactic structures can be
leveraged in the encoding stage through the integration of graph-based and
sequential models, and how multiple, specialised latent representations can be
injected into the decoder's attention mechanism via low-rank operators. Our
empirical evaluation, carried out on natural language sentences and
mathematical expressions, reveals that the proposed end-to-end VAE architecture
can result in a better overall organisation of the latent space, alleviating
the information loss occurring in standard VAE setups, resulting in enhanced
performances on language modelling and downstream generation tasks.
