  It has been proven that transfer learning provides an easy way to achieve
state-of-the-art accuracies on several vision tasks by training a simple
classifier on top of features obtained from pre-trained neural networks. The
goal of this work is to generate better features for transfer learning from
multiple publicly available pre-trained neural networks. To this end, we
propose a novel architecture called Stacked Neural Networks which leverages the
fast training time of transfer learning while simultaneously being much more
accurate. We show that using a stacked NN architecture can result in up to 8%
improvements in accuracy over state-of-the-art techniques using only one
pre-trained network for transfer learning. A second aim of this work is to make
network fine- tuning retain the generalizability of the base network to unseen
tasks. To this end, we propose a new technique called "joint fine-tuning" that
is able to give accuracies comparable to finetuning the same network
individually over two datasets. We also show that a jointly finetuned network
generalizes better to unseen tasks when compared to a network finetuned over a
single task.
