  Retriever-reader models achieve competitive performance across many different
NLP tasks such as open question answering and dialogue conversations. In this
work, we notice these models easily overfit the top-rank retrieval passages and
standard training fails to reason over the entire retrieval passages. We
introduce a learnable passage mask mechanism which desensitizes the impact from
the top-rank retrieval passages and prevents the model from overfitting.
Controlling the gradient variance with fewer mask candidates and selecting the
mask candidates with one-shot bi-level optimization, our learnable
regularization strategy enforces the answer generation to focus on the entire
retrieval passages. Experiments on different tasks across open question
answering, dialogue conversation, and fact verification show that our method
consistently outperforms its baselines. Extensive experiments and ablation
studies demonstrate that our method can be general, effective, and beneficial
for many NLP tasks.
