  This paper explores a new research problem of unsupervised transfer learning
across multiple spatiotemporal prediction tasks. Unlike most existing transfer
learning methods that focus on fixing the discrepancy between supervised tasks,
we study how to transfer knowledge from a zoo of unsupervisedly learned models
towards another predictive network. Our motivation is that models from
different sources are expected to understand the complex spatiotemporal
dynamics from different perspectives, thereby effectively supplementing the new
task, even if the task has sufficient training samples. Technically, we propose
a differentiable framework named transferable memory. It adaptively distills
knowledge from a bank of memory states of multiple pretrained RNNs, and applies
it to the target network via a novel recurrent structure called the
Transferable Memory Unit (TMU). Compared with finetuning, our approach yields
significant improvements on three benchmarks for spatiotemporal prediction, and
benefits the target task even from less relevant pretext ones.
