  This paper presents a novel training method, Conditional Masked Language
Modeling (CMLM), to effectively learn sentence representations on large scale
unlabeled corpora. CMLM integrates sentence representation learning into MLM
training by conditioning on the encoded vectors of adjacent sentences. Our
English CMLM model achieves state-of-the-art performance on SentEval, even
outperforming models learned using supervised signals. As a fully unsupervised
learning method, CMLM can be conveniently extended to a broad range of
languages and domains. We find that a multilingual CMLM model co-trained with
bitext retrieval (BR) and natural language inference (NLI) tasks outperforms
the previous state-of-the-art multilingual models by a large margin, e.g. 10%
improvement upon baseline models on cross-lingual semantic search. We explore
the same language bias of the learned representations, and propose a simple,
post-training and model agnostic approach to remove the language identifying
information from the representation while still retaining sentence semantics.
