  Bayesian neural network posterior distributions have a great number of modes
that correspond to the same network function. The abundance of such modes can
make it difficult for approximate inference methods to do their job. Recent
work has demonstrated the benefits of partial stochasticity for approximate
inference in Bayesian neural networks; inference can be less costly and
performance can sometimes be improved. I propose a structured way to select the
deterministic subset of weights that removes neuron permutation symmetries, and
therefore the corresponding redundant posterior modes. With a drastically
simplified posterior distribution, the performance of existing approximate
inference schemes is found to be greatly improved.
