  We study the practical consequences of dataset sampling strategies on the
ranking performance of recommendation algorithms. Recommender systems are
generally trained and evaluated on samples of larger datasets. Samples are
often taken in a naive or ad-hoc fashion: e.g. by sampling a dataset randomly
or by selecting users or items with many interactions. As we demonstrate,
commonly-used data sampling schemes can have significant consequences on
algorithm performance. Following this observation, this paper makes three main
contributions: (1) characterizing the effect of sampling on algorithm
performance, in terms of algorithm and dataset characteristics (e.g. sparsity
characteristics, sequential dynamics, etc.); (2) designing SVP-CF, which is a
data-specific sampling strategy, that aims to preserve the relative performance
of models after sampling, and is especially suited to long-tailed interaction
data; and (3) developing an oracle, Data-Genie, which can suggest the sampling
scheme that is most likely to preserve model performance for a given dataset.
The main benefit of Data-Genie is that it will allow recommender system
practitioners to quickly prototype and compare various approaches, while
remaining confident that algorithm performance will be preserved, once the
algorithm is retrained and deployed on the complete data. Detailed experiments
show that using Data-Genie, we can discard upto 5x more data than any sampling
strategy with the same level of performance.
