  A number of studies have shown that increasing the depth or width of
convolutional networks is a rewarding approach to improve the performance of
image recognition. In our study, however, we observed difficulties along both
directions. On one hand, the pursuit for very deep networks is met with a
diminishing return and increased training difficulty; on the other hand,
widening a network would result in a quadratic growth in both computational
cost and memory demand. These difficulties motivate us to explore structural
diversity in designing deep networks, a new dimension beyond just depth and
width. Specifically, we present a new family of modules, namely the
PolyInception, which can be flexibly inserted in isolation or in a composition
as replacements of different parts of a network. Choosing PolyInception modules
with the guidance of architectural efficiency can improve the expressive power
while preserving comparable computational cost. The Very Deep PolyNet, designed
following this direction, demonstrates substantial improvements over the
state-of-the-art on the ILSVRC 2012 benchmark. Compared to Inception-ResNet-v2,
it reduces the top-5 validation error on single crops from 4.9% to 4.25%, and
that on multi-crops from 3.7% to 3.45%.
