  Collaborative filtering is widely used in modern recommender systems. Recent
research shows that variational autoencoders (VAEs) yield state-of-the-art
performance by integrating flexible representations from deep neural networks
into latent variable models, mitigating limitations of traditional linear
factor models. VAEs are typically trained by maximizing the likelihood (MLE) of
users interacting with ground-truth items. While simple and often effective,
MLE-based training does not directly maximize the recommendation-quality
metrics one typically cares about, such as top-N ranking. In this paper we
investigate new methods for training collaborative filtering models based on
actor-critic reinforcement learning, to directly optimize the
non-differentiable quality metrics of interest. Specifically, we train a critic
network to approximate ranking-based metrics, and then update the actor network
(represented here by a VAE) to directly optimize against the learned metrics.
In contrast to traditional learning-to-rank methods that require to re-run the
optimization procedure for new lists, our critic-based method amortizes the
scoring process with a neural network, and can directly provide the
(approximate) ranking scores for new lists. Empirically, we show that the
proposed methods outperform several state-of-the-art baselines, including
recently-proposed deep learning approaches, on three large-scale real-world
datasets. The code to reproduce the experimental results and figure plots is on
Github: https://github.com/samlobel/RaCT_CF
