  Convolutional neural networks typically encode an input image into a series
of intermediate features with decreasing resolutions. While this structure is
suited to classification tasks, it does not perform well for tasks requiring
simultaneous recognition and localization (e.g., object detection). The
encoder-decoder architectures are proposed to resolve this by applying a
decoder network onto a backbone model designed for classification tasks. In
this paper, we argue encoder-decoder architecture is ineffective in generating
strong multi-scale features because of the scale-decreased backbone. We propose
SpineNet, a backbone with scale-permuted intermediate features and cross-scale
connections that is learned on an object detection task by Neural Architecture
Search. Using similar building blocks, SpineNet models outperform ResNet-FPN
models by ~3% AP at various scales while using 10-20% fewer FLOPs. In
particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and
achieves 52.1% AP with a RetinaNet detector on COCO for a single model without
test-time augmentation, significantly outperforms prior art of detectors.
SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy
improvement on a challenging iNaturalist fine-grained dataset. Code is at:
https://github.com/tensorflow/tpu/tree/master/models/official/detection.
