  Though offering amazing contextualized token-level representations, current
pre-trained language models take less attention on accurately acquiring
sentence-level representation during their self-supervised pre-training.
However, contrastive objectives which dominate the current sentence
representation learning bring little linguistic interpretability and no
performance guarantee on downstream semantic tasks. We instead propose a novel
generative self-supervised learning objective based on phrase reconstruction.
To overcome the drawbacks of previous generative methods, we carefully model
intra-sentence structure by breaking down one sentence into pieces of important
phrases. Empirical studies show that our generative learning achieves powerful
enough performance improvement and outperforms the current state-of-the-art
contrastive methods not only on the STS benchmarks, but also on downstream
semantic retrieval and reranking tasks. Our code is available at
https://github.com/chengzhipanpan/PaSeR.
