  Recently, the emergence of large language models (LLMs) has revolutionized
the paradigm of information retrieval (IR) applications, especially in web
search, by generating vast amounts of human-like texts on the Internet. As a
result, IR systems in the LLM era are facing a new challenge: the indexed
documents are now not only written by human beings but also automatically
generated by the LLMs. How these LLM-generated documents influence the IR
systems is a pressing and still unexplored question. In this work, we conduct a
quantitative evaluation of IR models in scenarios where both human-written and
LLM-generated texts are involved. Surprisingly, our findings indicate that
neural retrieval models tend to rank LLM-generated documents higher. We refer
to this category of biases in neural retrievers towards the LLM-generated
content as the \textbf{source bias}. Moreover, we discover that this bias is
not confined to the first-stage neural retrievers, but extends to the
second-stage neural re-rankers. Then, in-depth analyses from the perspective of
text compression indicate that LLM-generated texts exhibit more focused
semantics with less noise, making it easier for neural retrieval models to
semantic match. To mitigate the source bias, we also propose a plug-and-play
debiased constraint for the optimization objective, and experimental results
show its effectiveness. Finally, we discuss the potential severe concerns
stemming from the observed source bias and hope our findings can serve as a
critical wake-up call to the IR community and beyond. To facilitate future
explorations of IR in the LLM era, the constructed two new benchmarks are
available at https://github.com/KID-22/Source-Bias.
