  Though it has been recognized that recommending serendipitous (i.e.,
surprising and relevant) items can be helpful for increasing users'
satisfaction and behavioral intention, how to measure serendipity in the
offline environment is still an open issue. In recent years, a number of
metrics have been proposed, but most of them were based on researchers'
assumptions due to the serendipity's subjective nature. In order to validate
these metrics' actual performance, we collected over 10,000 users' real
feedback data and compared with the metrics' results. It turns out the user
profile based metrics, especially content-based ones, perform better than those
based on item popularity, in terms of estimating the unexpectedness facet of
recommendations. Moreover, the full metrics, which involve the unexpectedness
component, relevance, timeliness, and user curiosity, can more accurately
indicate the recommendation's serendipity degree, relative to those that just
involve some of them. The application of these metrics to several recommender
algorithms further consolidates their practical usage, because the comparison
results are consistent with those from user evaluation. Thus, this work is
constructive for filling the gap between offline measurement and user study on
recommendation serendipity.
