  Do convolutional networks really need a fixed feed-forward structure? What
if, after identifying the high-level concept of an image, a network could move
directly to a layer that can distinguish fine-grained differences? Currently, a
network would first need to execute sometimes hundreds of intermediate layers
that specialize in unrelated aspects. Ideally, the more a network already knows
about an image, the better it should be at deciding which layer to compute
next. In this work, we propose convolutional networks with adaptive inference
graphs (ConvNet-AIG) that adaptively define their network topology conditioned
on the input image. Following a high-level structure similar to residual
networks (ResNets), ConvNet-AIG decides for each input image on the fly which
layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns
distinct inference graphs for different categories. Both ConvNet-AIG with 50
and 101 layers outperform their ResNet counterpart, while using 20% and 38%
less computations respectively. By grouping parameters into layers for related
classes and only executing relevant layers, ConvNet-AIG improves both
efficiency and overall classification quality. Lastly, we also study the effect
of adaptive inference graphs on the susceptibility towards adversarial
examples. We observe that ConvNet-AIG shows a higher robustness than ResNets,
complementing other known defense mechanisms.
