  Despite the recent success of stochastic gradient descent in deep learning,
it is often difficult to train a deep neural network with an inappropriate
choice of its initial parameters. Even if training is successful, it has been
known that the initial parameter configuration may negatively impact
generalization. In this paper, we propose an unsupervised algorithm to find
good initialization for input data, given that a downstream task is d-way
classification. We first notice that each parameter configuration in the
parameter space corresponds to one particular downstream task of d-way
classification. We then conjecture that the success of learning is directly
related to how diverse downstream tasks are in the vicinity of the initial
parameters. We thus design an algorithm that encourages small perturbation to
the initial parameter configuration leads to a diverse set of d-way
classification tasks. In other words, the proposed algorithm ensures a solution
to any downstream task to be near the initial parameter configuration. We
empirically evaluate the proposed algorithm on various tasks derived from MNIST
with a fully connected network. In these experiments, we observe that our
algorithm improves average test accuracy across most of these tasks, and that
such improvement is greater when the number of labelled examples is small.
