  There have been many successful applications of sentence embedding methods.
However, it has not been well understood what properties are captured in the
resulting sentence embeddings depending on the supervision signals. In this
paper, we focus on two types of sentence embedding methods with similar
architectures and tasks: one fine-tunes pre-trained language models on the
natural language inference task, and the other fine-tunes pre-trained language
models on word prediction task from its definition sentence, and investigate
their properties. Specifically, we compare their performances on semantic
textual similarity (STS) tasks using STS datasets partitioned from two
perspectives: 1) sentence source and 2) superficial similarity of the sentence
pairs, and compare their performances on the downstream and probing tasks.
Furthermore, we attempt to combine the two methods and demonstrate that
combining the two methods yields substantially better performance than the
respective methods on unsupervised STS tasks and downstream tasks.
