  In session-based or sequential recommendation, it is important to consider a
number of factors like long-term user engagement, multiple types of user-item
interactions such as clicks, purchases etc. The current state-of-the-art
supervised approaches fail to model them appropriately. Casting sequential
recommendation task as a reinforcement learning (RL) problem is a promising
direction. A major component of RL approaches is to train the agent through
interactions with the environment. However, it is often problematic to train a
recommender in an on-line fashion due to the requirement to expose users to
irrelevant recommendations. As a result, learning the policy from logged
implicit feedback is of vital importance, which is challenging due to the pure
off-policy setting and lack of negative rewards (feedback). In this paper, we
propose self-supervised reinforcement learning for sequential recommendation
tasks. Our approach augments standard recommendation models with two output
layers: one for self-supervised learning and the other for RL. The RL part acts
as a regularizer to drive the supervised layer focusing on specific
rewards(e.g., recommending items which may lead to purchases rather than
clicks) while the self-supervised layer with cross-entropy loss provides strong
gradient signals for parameter updates. Based on such an approach, we propose
two frameworks namely Self-Supervised Q-learning(SQN) and Self-Supervised
Actor-Critic(SAC). We integrate the proposed frameworks with four
state-of-the-art recommendation models. Experimental results on two real-world
datasets demonstrate the effectiveness of our approach.
