  The encoder-decoder models for unsupervised sentence representation learning
tend to discard the decoder after being trained on a large unlabelled corpus,
since only the encoder is needed to map the input sentence into a vector
representation. However, parameters learnt in the decoder also contain useful
information about language. In order to utilise the decoder after learning, we
present two types of decoding functions whose inverse can be easily derived
without expensive inverse calculation. Therefore, the inverse of the decoding
function serves as another encoder that produces sentence representations. We
show that, with careful design of the decoding functions, the model learns good
sentence representations, and the ensemble of the representations produced from
the encoder and the inverse of the decoder demonstrate even better
generalisation ability and solid transferability.
