  Offline evaluations of recommender systems attempt to estimate users'
satisfaction with recommendations using static data from prior user
interactions. These evaluations provide researchers and developers with first
approximations of the likely performance of a new system and help weed out bad
ideas before presenting them to users. However, offline evaluation cannot
accurately assess novel, relevant recommendations, because the most novel items
were previously unknown to the user, so they are missing from the historical
data and cannot be judged as relevant.
  We present a simulation study to estimate the error that such missing data
causes in commonly-used evaluation metrics in order to assess its prevalence
and impact. We find that missing data in the rating or observation process
causes the evaluation protocol to systematically mis-estimate metric values,
and in some cases erroneously determine that a popularity-based recommender
outperforms even a perfect personalized recommender. Substantial breakthroughs
in recommendation quality, therefore, will be difficult to assess with existing
offline techniques.
