  Establishing a docker-based replicability infrastructure offers the community
a great opportunity: measuring the run time of information retrieval systems.
The time required to present query results to a user is paramount to the users
satisfaction. Recent advances in neural IR re-ranking models put the issue of
query latency at the forefront. They bring a complex trade-off between
performance and effectiveness based on a myriad of factors: the choice of
encoding model, network architecture, hardware acceleration and many others.
The best performing models (currently using the BERT transformer model) run
orders of magnitude more slowly than simpler architectures. We aim to broaden
the focus of the neural IR community to include performance considerations --
to sustain the practical applicability of our innovations. In this position
paper we supply our argument with a case study exploring the performance of
different neural re-ranking models. Finally, we propose to extend the OSIRRC
docker-based replicability infrastructure with two performance focused
benchmark scenarios.
