  SegBlocks reduces the computational cost of existing neural networks, by
dynamically adjusting the processing resolution of image regions based on their
complexity. Our method splits an image into blocks and downsamples blocks of
low complexity, reducing the number of operations and memory consumption. A
lightweight policy network, selecting the complex regions, is trained using
reinforcement learning. In addition, we introduce several modules implemented
in CUDA to process images in blocks. Most important, our novel BlockPad module
prevents the feature discontinuities at block borders of which existing methods
suffer, while keeping memory consumption under control. Our experiments on
Cityscapes, Camvid and Mapillary Vistas datasets for semantic segmentation show
that dynamically processing images offers a better accuracy versus complexity
trade-off compared to static baselines of similar complexity. For instance, our
method reduces the number of floating-point operations of SwiftNet-RN18 by 60%
and increases the inference speed by 50%, with only 0.3% decrease in mIoU
accuracy on Cityscapes.
