  Large width limits have been a recent focus of deep learning research: modulo
computational practicalities, do wider networks outperform narrower ones?
Answering this question has been challenging, as conventional networks gain
representational power with width, potentially masking any negative effects.
Our analysis in this paper decouples capacity and width via the generalization
of neural networks to Deep Gaussian Processes (Deep GP), a class of
nonparametric hierarchical models that subsume neural nets. In doing so, we aim
to understand how width affects (standard) neural networks once they have
sufficient capacity for a given modeling task. Our theoretical and empirical
results on Deep GP suggest that large width can be detrimental to hierarchical
models. Surprisingly, we prove that even nonparametric Deep GP converge to
Gaussian processes, effectively becoming shallower without any increase in
representational power. The posterior, which corresponds to a mixture of
data-adaptable basis functions, becomes less data-dependent with width. Our
tail analysis demonstrates that width and depth have opposite effects: depth
accentuates a model's non-Gaussianity, while width makes models increasingly
Gaussian. We find there is a "sweet spot" that maximizes test performance
before the limiting GP behavior prevents adaptability, occurring at width = 1
or width = 2 for nonparametric Deep GP. These results make strong predictions
about the same phenomenon in conventional neural networks trained with L2
regularization (analogous to a Gaussian prior on parameters): we show that such
neural networks may need up to 500 - 1000 hidden units for sufficient capacity
- depending on the dataset - but further width degrades performance.
