  In this study, we address the challenge of measuring the ability of a
recommender system to make surprising recommendations. Although current
evaluation methods make it possible to determine if two algorithms can make
recommendations with a significant difference in their average surprise
measure, it could be of interest to our community to know how competent an
algorithm is at embedding surprise in its recommendations, without having to
resort to making a direct comparison with another algorithm. We argue that a)
surprise is a finite resource in a recommender system, b) there is a limit to
how much surprise any algorithm can embed in a recommendation, and c) this
limit can provide us with a scale against which the performance of any
algorithm can be measured. By exploring these ideas, it is possible to define
the concepts of maximum and minimum potential surprise and design a surprise
metric called "normalised surprise" that employs these limits to potential
surprise. Two experiments were conducted to test the proposed metric. The aim
of the first was to validate the quality of the estimates of minimum and
maximum potential surprise produced by a greedy algorithm. The purpose of the
second experiment was to analyse the behaviour of the proposed metric using the
MovieLens dataset. The results confirmed the behaviour that was expected, and
showed that the proposed surprise metric is both effective and consistent for
differing choices of recommendation algorithms, data representations and
distance functions.
