  While other areas of machine learning have seen more and more automation,
designing a high-performing recommender system still requires a high level of
human effort. Furthermore, recent work has shown that modern recommender system
algorithms do not always improve over well-tuned baselines. A natural follow-up
question is, "how do we choose the right algorithm for a new dataset and
performance metric?" In this work, we start by giving the first large-scale
study of recommender system approaches by comparing 18 algorithms and 100 sets
of hyperparameters across 85 datasets and 315 metrics. We find that the best
algorithms and hyperparameters are highly dependent on the dataset and
performance metric, however, there are also strong correlations between the
performance of each algorithm and various meta-features of the datasets.
Motivated by these findings, we create RecZilla, a meta-learning approach to
recommender systems that uses a model to predict the best algorithm and
hyperparameters for new, unseen datasets. By using far more meta-training data
than prior work, RecZilla is able to substantially reduce the level of human
involvement when faced with a new recommender system application. We not only
release our code and pretrained RecZilla models, but also all of our raw
experimental results, so that practitioners can train a RecZilla model for
their desired performance metric: https://github.com/naszilla/reczilla.
