  Transfer learning methods start performing poorly when the complexity of the
learning task is increased. Most of these methods calculate the cumulative
differences of all the matched features and then use them to back-propagate
that loss through all the layers. Contrary to these methods, in this work, we
propose a novel layer-wise learning scheme that adjusts learning parameters per
layer as a function of the differences in the Jacobian/Attention/Hessian of the
output activations w.r.t. the network parameters. We applied this novel scheme
for attention map-based and derivative-based (first and second order) transfer
learning methods. We received improved learning performance and stability
against a wide range of datasets. From extensive experimental evaluation, we
observed that the performance boost achieved by our method becomes more
significant with the increasing difficulty of the learning task.
