  Recommender models are hard to evaluate, particularly under offline setting.
In this paper, we provide a comprehensive and critical analysis of the data
leakage issue in recommender system offline evaluation. Data leakage is caused
by not observing global timeline in evaluating recommenders, e.g., train/test
data split does not follow global timeline. As a result, a model learns from
the user-item interactions that are not expected to be available at prediction
time. We first show the temporal dynamics of user-item interactions along
global timeline, then explain why data leakage exists for collaborative
filtering models. Through carefully designed experiments, we show that all
models indeed recommend future items that are not available at the time point
of a test instance, as the result of data leakage. The experiments are
conducted with four widely used baseline models - BPR, NeuMF, SASRec, and
LightGCN, on four popular offline datasets - MovieLens-25M, Yelp, Amazon-music,
and Amazon-electronic, adopting leave-last-one-out data split. We further show
that data leakage does impact models' recommendation accuracy. Their relative
performance orders thus become unpredictable with different amount of leaked
future data in training. To evaluate recommendation systems in a realistic
manner in offline setting, we propose a timeline scheme, which calls for a
revisit of the recommendation model design.
