  Text variational autoencoders (VAEs) are notorious for posterior collapse, a
phenomenon where the model's decoder learns to ignore signals from the encoder.
Because posterior collapse is known to be exacerbated by expressive decoders,
Transformers have seen limited adoption as components of text VAEs. Existing
studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et
al., 2021) mitigate posterior collapse using massive pretraining, a technique
unavailable to most of the research community without extensive computing
resources. We present a simple two-phase training scheme to convert a
sequence-to-sequence Transformer into a VAE with just finetuning. The resulting
language model is competitive with massively pretrained Transformer-based VAEs
in some internal metrics while falling short on others. To facilitate training
we comprehensively explore the impact of common posterior collapse alleviation
techniques in the literature. We release our code for reproducability.
