  Imagine you are a teacher attempting to assess a student's level in a
particular subject. If you design a test with only hard questions, and the
student fails, this mostly proves that the student does not understand the more
advanced material. A more insightful exam would include different types of
questions varying in difficulty to truly understand the student's weaknesses
and strengths from different perspectives. In the field of Recommender Systems
(RS), more often than not, we design evaluations to measure an algorithm's
ability to optimize goals in complex scenarios, representative of the
real-world challenges the system would most probably face. Nevertheless, this
paper posits that testing an algorithm's ability to address both simple and
complex tasks/problems would offer a more detailed view of performance to help
identify, at a more granular level, the weaknesses and strengths of solutions
when facing different scenarios/domains. We believe the RS community would
greatly benefit from creating a collection of standardized, simple, and
targeted experiments, which, much like a suite of "unit tests", would
individually assess an algorithm's ability to tackle core challenges that make
up complex RS tasks. What's more, these experiments go beyond traditional
pass/fail "unit tests". Running an algorithm against the collection of
experiments allows a researcher to empirically analyze in which type of
settings an algorithm performs best and to what degree under different metrics.
Not only do we defend this position, in this paper, we also offer a proposal of
how these simple and targeted experiments could be defined and shared and
suggest potential next steps to make this project a reality.
