  The progress of recommender systems is hampered mainly by evaluation as it
requires real-time interactions between humans and systems, which is too
laborious and expensive. This issue is usually approached by utilizing the
interaction history to conduct offline evaluation. However, existing datasets
of user-item interactions are partially observed, leaving it unclear how and to
what extent the missing interactions will influence the evaluation. To answer
this question, we collect a fully-observed dataset from Kuaishou's online
environment, where almost all 1,411 users have been exposed to all 3,327 items.
To the best of our knowledge, this is the first real-world fully-observed data
with millions of user-item interactions.
  With this unique dataset, we conduct a preliminary analysis of how the two
factors - data density and exposure bias - affect the evaluation results of
multi-round conversational recommendation. Our main discoveries are that the
performance ranking of different methods varies with the two factors, and this
effect can only be alleviated in certain cases by estimating missing
interactions for user simulation. This demonstrates the necessity of the
fully-observed dataset. We release the dataset and the pipeline implementation
for evaluation at https://kuairec.com
