  Recommender systems research tends to evaluate model performance offline and
on randomly sampled targets, yet the same systems are later used to predict
user behavior sequentially from a fixed point in time. Simulating online
recommender system performance is notoriously difficult and the discrepancy
between online and offline behaviors is typically not accounted for in offline
evaluations. This disparity permits weaknesses to go unnoticed until the model
is deployed in a production setting. In this paper, we first demonstrate how
omitting temporal context when evaluating recommender system performance leads
to false confidence. To overcome this, we postulate that offline evaluation
protocols can only model real-life use-cases if they account for temporal
context. Next, we propose a training procedure to further embed the temporal
context in existing models. We use a multi-objective approach to introduce
temporal context into traditionally time-unaware recommender systems and
confirm its advantage via the proposed evaluation protocol. Finally, we
validate that the Pareto Fronts obtained with the added objective dominate
those produced by state-of-the-art models that are only optimized for accuracy
on three real-world publicly available datasets. The results show that
including our temporal objective can improve recall@20 by up to 20%.
