  In this paper we propose a Bayesian method for estimating architectural
parameters of neural networks, namely layer size and network depth. We do this
by learning concrete distributions over these parameters. Our results show that
regular networks with a learnt structure can generalise better on small
datasets, while fully stochastic networks can be more robust to parameter
initialisation. The proposed method relies on standard neural variational
learning and, unlike randomised architecture search, does not require a
retraining of the model, thus keeping the computational overhead at minimum.
