  We consider $Q$-learning with knowledge transfer, using samples from a target
reinforcement learning (RL) task as well as source samples from different but
related RL tasks. We propose transfer learning algorithms for both batch and
online $Q$-learning with offline source studies. The proposed transferred
$Q$-learning algorithm contains a novel re-targeting step that enables vertical
information-cascading along multiple steps in an RL task, besides the usual
horizontal information-gathering as transfer learning (TL) for supervised
learning. We establish the first theoretical justifications of TL in RL tasks
by showing a faster rate of convergence of the $Q$ function estimation in the
offline RL transfer, and a lower regret bound in the offline-to-online RL
transfer under certain similarity assumptions. Empirical evidences from both
synthetic and real datasets are presented to back up the proposed algorithm and
our theoretical results.
