  Sentence embedding is an important research topic in natural language
processing (NLP) since it can transfer knowledge to downstream tasks.
Meanwhile, a contextualized word representation, called BERT, achieves the
state-of-the-art performance in quite a few NLP tasks. Yet, it is an open
problem to generate a high quality sentence representation from BERT-based word
models. It was shown in previous study that different layers of BERT capture
different linguistic properties. This allows us to fusion information across
layers to find better sentence representation. In this work, we study the
layer-wise pattern of the word representation of deep contextualized models.
Then, we propose a new sentence embedding method by dissecting BERT-based word
models through geometric analysis of the space spanned by the word
representation. It is called the SBERT-WK method. No further training is
required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and
downstream supervised tasks. Furthermore, ten sentence-level probing tasks are
presented for detailed linguistic analysis. Experiments show that SBERT-WK
achieves the state-of-the-art performance. Our codes are publicly available.
