  Even though offline evaluation is just an imperfect proxy of online
performance -- due to the interactive nature of recommenders -- it will
probably remain the primary way of evaluation in recommender systems research
for the foreseeable future, since the proprietary nature of production
recommenders prevents independent validation of A/B test setups and
verification of online results. Therefore, it is imperative that offline
evaluation setups are as realistic and as flawless as they can be.
Unfortunately, evaluation flaws are quite common in recommender systems
research nowadays, due to later works copying flawed evaluation setups from
their predecessors without questioning their validity. In the hope of improving
the quality of offline evaluation of recommender systems, we discuss four of
these widespread flaws and why researchers should avoid them.
