  Passage retrieval aims to retrieve relevant passages from large collections
of the open-domain corpus. Contextual Masked Auto-Encoding has been proven
effective in representation bottleneck pre-training of a monolithic
dual-encoder for passage retrieval. Siamese or fully separated dual-encoders
are often adopted as basic retrieval architecture in the pre-training and
fine-tuning stages for encoding queries and passages into their latent
embedding spaces. However, simply sharing or separating the parameters of the
dual-encoder results in an imbalanced discrimination of the embedding spaces.
In this work, we propose to pre-train Contextual Masked Auto-Encoder with
Mixture-of-Textual-Experts (CoT-MoTE). Specifically, we incorporate
textual-specific experts for individually encoding the distinct properties of
queries and passages. Meanwhile, a shared self-attention layer is still kept
for unified attention modeling. Results on large-scale passage retrieval
benchmarks show steady improvement in retrieval performances. The quantitive
analysis also shows a more balanced discrimination of the latent embedding
spaces.
