  It is common for video-on-demand and music streaming services to adopt a user
interface composed of several recommendation lists, i.e. widgets or swipeable
carousels, each generated according to a specific criterion or algorithm (e.g.
most recent, top popular, recommended for you, editors' choice, etc.).
Selecting the appropriate combination of carousel has significant impact on
user satisfaction. A crucial aspect of this user interface is that to measure
the relevance a new carousel for the user it is not sufficient to account
solely for its individual quality. Instead, it should be considered that other
carousels will already be present in the interface. This is not considered by
traditional evaluation protocols for recommenders systems, in which each
carousel is evaluated in isolation, regardless of (i) which other carousels are
displayed to the user and (ii) the relative position of the carousel with
respect to other carousels. Hence, we propose a two-dimensional evaluation
protocol for a carousel setting that will measure the quality of a
recommendation carousel based on how much it improves upon the quality of an
already available set of carousels. Our evaluation protocol takes into account
also the position bias, i.e. users do not explore the carousels sequentially,
but rather concentrate on the top-left corner of the screen.
  We report experiments on the movie domain and notice that under a carousel
setting the definition of which criteria has to be preferred to generate a list
of recommended items changes with respect to what is commonly understood.
