  Dense retrieval has shown promising results in many information retrieval
(IR) related tasks, whose foundation is high-quality text representation
learning for effective search. Some recent studies have shown that
autoencoder-based language models are able to boost the dense retrieval
performance using a weak decoder. However, we argue that 1) it is not
discriminative to decode all the input texts and, 2) even a weak decoder has
the bypass effect on the encoder. Therefore, in this work, we introduce a novel
contrastive span prediction task to pre-train the encoder alone, but still
retain the bottleneck ability of the autoencoder. % Therefore, in this work, we
propose to drop out the decoder and introduce a novel contrastive span
prediction task to pre-train the encoder alone. The key idea is to force the
encoder to generate the text representation close to its own random spans while
far away from others using a group-wise contrastive loss. In this way, we can
1) learn discriminative text representations efficiently with the group-wise
contrastive learning over spans and, 2) avoid the bypass effect of the decoder
thoroughly. Comprehensive experiments over publicly available retrieval
benchmark datasets show that our approach can outperform existing pre-training
methods for dense retrieval significantly.
