  Reinforcement learning (RL) is an effective method of finding reasoning
pathways in incomplete knowledge graphs (KGs). To overcome the challenges of a
large action space, a self-supervised pre-training method is proposed to warm
up the policy network before the RL training stage. To alleviate the
distributional mismatch issue in general self-supervised RL (SSRL), in our
supervised learning (SL) stage, the agent selects actions based on the policy
network and learns from generated labels; this self-generation of labels is the
intuition behind the name self-supervised. With this training framework, the
information density of our SL objective is increased and the agent is prevented
from getting stuck with the early rewarded paths. Our self-supervised RL (SSRL)
method improves the performance of RL by pairing it with the wide coverage
achieved by SL during pretraining, since the breadth of the SL objective makes
it infeasible to train an agent with that alone. We show that our SSRL model
meets or exceeds current state-of-the-art results on all Hits@k and mean
reciprocal rank (MRR) metrics on four large benchmark KG datasets. This SSRL
method can be used as a plug-in for any RL architecture for a KGR task. We
adopt two RL architectures, i.e., MINERVA and MultiHopKG as our baseline RL
models and experimentally show that our SSRL model consistently outperforms
both baselines on all of these four KG reasoning tasks. Full code for the paper
available at
https://github.com/owenonline/Knowledge-Graph-Reasoning-with-Self-supervised-Reinforcement-Learning.
