  Evaluation in Information Retrieval relies on post-hoc empirical procedures,
which are time-consuming and expensive operations. To alleviate this, Query
Performance Prediction (QPP) models have been developed to estimate the
performance of a system without the need for human-made relevance judgements.
Such models, usually relying on lexical features from queries and corpora, have
been applied to traditional sparse IR methods - with various degrees of
success. With the advent of neural IR and large Pre-trained Language Models,
the retrieval paradigm has significantly shifted towards more semantic signals.
In this work, we study and analyze to what extent current QPP models can
predict the performance of such systems. Our experiments consider seven
traditional bag-of-words and seven BERT-based IR approaches, as well as
nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning '19
and Robust '04. Our findings show that QPPs perform statistically significantly
worse on neural IR systems. In settings where semantic signals are prominent
(e.g., passage retrieval), their performance on neural models drops by as much
as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented
scenarios, QPPs fail to predict performance for neural IR systems on those
queries where they differ from traditional approaches the most.
