  Mean field theory has been successfully used to analyze deep neural networks
(DNN) in the infinite size limit. Given the finite size of realistic DNN, we
utilize the large deviation theory and path integral analysis to study the
deviation of functions represented by DNN from their typical mean field
solutions. The parameter perturbations investigated include weight
sparsification (dilution) and binarization, which are commonly used in model
simplification, for both ReLU and sign activation functions. We find that
random networks with ReLU activation are more robust to parameter perturbations
with respect to their counterparts with sign activation, which arguably is
reflected in the simplicity of the functions they generate.
