  Formal/symbolic semantics can provide canonical, rigid controllability and
interpretability to sentence representations due to their \textit{localisation}
or \textit{composition} property. How can we deliver such property to the
current distributional sentence representations to control and interpret the
generation of language models (LMs)? In this work, we theoretically frame the
sentence semantics as the composition of \textit{semantic role - word content}
features and propose the formal semantic geometry. To inject such geometry into
Transformer-based LMs (i.e. GPT2), we deploy Transformer-based Variational
AutoEncoder with a supervision approach, where the sentence generation can be
manipulated and explained over low-dimensional latent Gaussian space. In
addition, we propose a new probing algorithm to guide the movement of sentence
vectors over such geometry. Experimental results reveal that the formal
semantic geometry can potentially deliver better control and interpretation to
sentence generation.
