  Transfer learning in Reinforcement Learning (RL) has been widely studied to
overcome training issues of Deep-RL, i.e., exploration cost, data availability
and convergence time, by introducing a way to enhance training phase with
external knowledge. Generally, knowledge is transferred from expert-agents to
novices. While this fixes the issue for a novice agent, a good understanding of
the task on expert agent is required for such transfer to be effective. As an
alternative, in this paper we propose Expert-Free Online Transfer Learning
(EF-OnTL), an algorithm that enables expert-free real-time dynamic transfer
learning in multi-agent system. No dedicated expert exists, and transfer source
agent and knowledge to be transferred are dynamically selected at each transfer
step based on agents' performance and uncertainty. To improve uncertainty
estimation, we also propose State Action Reward Next-State Random Network
Distillation (sars-RND), an extension of RND that estimates uncertainty from RL
agent-environment interaction. We demonstrate EF-OnTL effectiveness against a
no-transfer scenario and advice-based baselines, with and without expert
agents, in three benchmark tasks: Cart-Pole, a grid-based Multi-Team
Predator-Prey (mt-pp) and Half Field Offense (HFO). Our results show that
EF-OnTL achieve overall comparable performance when compared against
advice-based baselines while not requiring any external input nor threshold
tuning. EF-OnTL outperforms no-transfer with an improvement related to the
complexity of the task addressed.
