  Research on recommender systems is a challenging task, as is building and
operating such systems. Major challenges include non-reproducible research
results, dealing with noisy data, and answering many questions such as how many
recommendations to display, how often, and, of course, how to generate
recommendations most effectively. In the past six years, we built three
research-article recommender systems for digital libraries and reference
managers, and conducted research on these systems. In this paper, we share some
experiences we made during that time. Among others, we discuss the required
skills to build recommender systems, and why the literature provides little
help in identifying promising recommendation approaches. We explain the
challenge in creating a randomization engine to run A/B tests, and how low data
quality impacts the calculation of bibliometrics. We further discuss why
several of our experiments delivered disappointing results, and provide
statistics on how many researchers showed interest in our recommendation
dataset.
