  We analyze the learning dynamics of infinitely wide neural networks with a
finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck
in an otherwise infinite width network al-lows data dependent feature learning
in its bottle-neck representation. We empirically show that a single bottleneck
in infinite networks dramatically accelerates training when compared to purely
in-finite networks, with an improved overall performance. We discuss the
acceleration phenomena by drawing similarities to infinitely wide deep linear
models, where the acceleration effect of a bottleneck can be understood
theoretically.
