  In recent years, it is common practice to extract fully-connected layer (fc)
features that were learned while performing image classification on a source
dataset, such as ImageNet, and apply them generally to a wide range of other
tasks. The general usefulness of some large training datasets for transfer
learning is not yet well understood, and raises a number of questions. For
example, in the context of transfer learning, what is the role of a specific
class in the source dataset, and how is the transferability of fc features
affected when they are trained using various subsets of the set of all classes
in the source dataset? In this paper, we address the question of how to select
an optimal subset of the set of classes, subject to a budget constraint, that
will more likely generate good features for other tasks. To accomplish this, we
use a submodular set function to model the accuracy achievable on a new task
when the features have been learned on a given subset of classes of the source
dataset. An optimal subset is identified as the set that maximizes this
submodular function. The maximization can be accomplished using an efficient
greedy algorithm that comes with guarantees on the optimality of the solution.
We empirically validate our submodular model by successfully identifying
subsets of classes that produce good features for new tasks.
