  Understanding and controlling biasing effects in neural networks is crucial
for ensuring accurate and fair model performance. In the context of
classification problems, we provide a theoretical analysis demonstrating that
the structure of a deep neural network (DNN) can condition the model to assign
all predictions to the same class, even before the beginning of training, and
in the absence of explicit biases. We prove that, besides dataset properties,
the presence of this phenomenon, which we call \textit{Initial Guessing Bias}
(IGB), is influenced by model choices including dataset preprocessing methods,
and architectural decisions, such as activation functions, max-pooling layers,
and network depth. Our analysis of IGB provides information for architecture
selection and model initialization. We also highlight theoretical consequences,
such as the breakdown of node-permutation symmetry, the violation of
self-averaging and the non-trivial effects that depth has on the phenomenon.
