  To better support retrieval applications such as web search and question
answering, growing effort is made to develop retrieval-oriented language
models. Most of the existing works focus on improving the semantic
representation capability for the contextualized embedding of [CLS] token.
However, recent study shows that the ordinary tokens besides [CLS] may provide
extra information, which helps to produce a better representation effect. As
such, it's necessary to extend the current methods where all contextualized
embeddings can be jointly pre-trained for the retrieval tasks.
  With this motivation, we propose a new pre-training method: duplex masked
auto-encoder, a.k.a. DupMAE, which targets on improving the semantic
representation capacity for the contextualized embeddings of both [CLS] and
ordinary tokens. It introduces two decoding tasks: one is to reconstruct the
original input sentence based on the [CLS] embedding, the other one is to
minimize the bag-of-words loss (BoW) about the input sentence based on the
entire ordinary tokens' embeddings. The two decoding losses are added up to
train a unified encoding model. The embeddings from [CLS] and ordinary tokens,
after dimension reduction and aggregation, are concatenated as one unified
semantic representation for the input. DupMAE is simple but empirically
competitive: with a small decoding cost, it substantially contributes to the
model's representation capability and transferability, where remarkable
improvements are achieved on MS MARCO and BEIR benchmarks.
