  Transfer learning is widely used in deep neural network models when there are
few labeled examples available. The common approach is to take a pre-trained
network in a similar task and finetune the model parameters. This is usually
done blindly without a pre-selection from a set of pre-trained models, or by
finetuning a set of models trained on different tasks and selecting the best
performing one by cross-validation. We address this problem by proposing an
approach to assess the relationship between visual tasks and their
task-specific models. Our method uses Representation Similarity Analysis (RSA),
which is commonly used to find a correlation between neuronal responses from
brain data and models. With RSA we obtain a similarity score among tasks by
computing correlations between models trained on different tasks. Our method is
efficient as it requires only pre-trained models, and a few images with no
further training. We demonstrate the effectiveness and efficiency of our method
for generating task taxonomy on Taskonomy dataset. We next evaluate the
relationship of RSA with the transfer learning performance on Taskonomy tasks
and a new task: Pascal VOC semantic segmentation. Our results reveal that
models trained on tasks with higher similarity score show higher transfer
learning performance. Surprisingly, the best transfer learning result for
Pascal VOC semantic segmentation is not obtained from the pre-trained model on
semantic segmentation, probably due to the domain differences, and our method
successfully selects the high performing models.
