  Neural ranking models (NRMs) have become one of the most important techniques
in information retrieval (IR). Due to the limitation of relevance labels, the
training of NRMs heavily relies on negative sampling over unlabeled data. In
general machine learning scenarios, it has shown that training with hard
negatives (i.e., samples that are close to positives) could lead to better
performance. Surprisingly, we find opposite results from our empirical studies
in IR. When sampling top-ranked results (excluding the labeled positives) as
negatives from a stronger retriever, the performance of the learned NRM becomes
even worse. Based on our investigation, the superficial reason is that there
are more false negatives (i.e., unlabeled positives) in the top-ranked results
with a stronger retriever, which may hurt the training process; The root is the
existence of pooling bias in the dataset constructing process, where annotators
only judge and label very few samples selected by some basic retrievers.
Therefore, in principle, we can formulate the false negative issue in training
NRMs as learning from labeled datasets with pooling bias. To solve this
problem, we propose a novel Coupled Estimation Technique (CET) that learns both
a relevance model and a selection model simultaneously to correct the pooling
bias for training NRMs. Empirical results on three retrieval benchmarks show
that NRMs trained with our technique can achieve significant gains on ranking
effectiveness against other baseline strategies.
