  Recommender systems nowadays have many applications and are of great economic
benefit. Hence, it is imperative for success-oriented companies to compare
different of such systems and select the better one for their purposes. To this
end, various metrics of predictive accuracy are commonly used, such as the Root
Mean Square Error (RMSE), or precision and recall. All these metrics more or
less measure how well a recommender system can predict human behaviour.
  Unfortunately, human behaviour is always associated with some degree of
uncertainty, making the evaluation difficult, since it is not clear whether a
deviation is system-induced or just originates from the natural variability of
human decision making. At this point, some authors speculated that we may be
reaching some Magic Barrier where this variability prevents us from getting
much more accurate.
  In this article, we will extend the existing theory of the Magic Barrier into
a new probabilistic but a yet pragmatic model. In particular, we will use
methods from metrology and physics to develop easy-to-handle quantities for
computation to describe the Magic Barrier for different accuracy metrics and
provide suggestions for common application. This discussion is substantiated by
comprehensive experiments with real users and large-scale simulations on a
high-performance cluster.
