  There has been a lot of recent interest in trying to characterize the error
surface of deep models. This stems from a long standing question. Given that
deep networks are highly nonlinear systems optimized by local gradient methods,
why do they not seem to be affected by bad local minima? It is widely believed
that training of deep models using gradient methods works so well because the
error surface either has no local minima, or if they exist they need to be
close in value to the global minimum. It is known that such results hold under
very strong assumptions which are not satisfied by real models. In this paper
we present examples showing that for such theorem to be true additional
assumptions on the data, initialization schemes and/or the model classes have
to be made. We look at the particular case of finite size datasets. We
demonstrate that in this scenario one can construct counter-examples (datasets
or initialization schemes) when the network does become susceptible to bad
local minima over the weight space.
