  In traditional recommender system literature, diversity is often seen as the
opposite of similarity, and typically defined as the distance between
identified topics, categories or word models. However, this is not expressive
of the social science's interpretation of diversity, which accounts for a news
organization's norms and values and which we here refer to as normative
diversity. We introduce RADio, a versatile metrics framework to evaluate
recommendations according to these normative goals. RADio introduces a
rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a
user's decreasing propensity to observe items further down a list and (ii) full
distributional shifts as opposed to point estimates. We evaluate RADio's
ability to reflect five normative concepts in news recommendations on the
Microsoft News Dataset and six (neural) recommendation algorithms, with the
help of our metadata enrichment pipeline. We find that RADio provides
insightful estimates that can potentially be used to inform news recommender
system design.
