  Modern large-scale recommender systems are built upon computation-intensive
infrastructure and usually suffer from a huge difference in traffic between
peak and off-peak periods. In peak periods, it is challenging to perform
real-time computation for each request due to the limited budget of
computational resources. The recommendation with a cache is a solution to this
problem, where a user-wise result cache is used to provide recommendations when
the recommender system cannot afford a real-time computation. However, the
cached recommendations are usually suboptimal compared to real-time
computation, and it is challenging to determine the items in the cache for each
user. In this paper, we provide a cache-aware reinforcement learning (CARL)
method to jointly optimize the recommendation by real-time computation and by
the cache. We formulate the problem as a Markov decision process with user
states and a cache state, where the cache state represents whether the
recommender system performs recommendations by real-time computation or by the
cache. The computational load of the recommender system determines the cache
state. We perform reinforcement learning based on such a model to improve user
engagement over multiple requests. Moreover, we show that the cache will
introduce a challenge called critic dependency, which deteriorates the
performance of reinforcement learning. To tackle this challenge, we propose an
eigenfunction learning (EL) method to learn independent critics for CARL.
Experiments show that CARL can significantly improve the users' engagement when
considering the result cache. CARL has been fully launched in Kwai app, serving
over 100 million users.
