  This paper addresses the task of semantic segmentation of orthoimagery using
multimodal data e.g. optical RGB, infrared and digital surface model. We
propose a deep convolutional neural network architecture termed OrthoSeg for
semantic segmentation using multimodal, orthorectified and coregistered data.
We also propose a training procedure for supervised training of OrthoSeg. The
training procedure complements the inherent architectural characteristics of
OrthoSeg for preventing complex co-adaptations of learned features, which may
arise due to probable high dimensionality and spatial correlation in multimodal
and/or multispectral coregistered data. OrthoSeg consists of parallel encoding
networks for independent encoding of multimodal feature maps and a decoder
designed for efficiently fusing independently encoded multimodal feature maps.
A softmax layer at the end of the network uses the features generated by the
decoder for pixel-wise classification. The decoder fuses feature maps from the
parallel encoders locally as well as contextually at multiple scales to
generate per-pixel feature maps for final pixel-wise classification resulting
in segmented output. We experimentally show the merits of OrthoSeg by
demonstrating state-of-the-art accuracy on the ISPRS Potsdam 2D Semantic
Segmentation dataset. Adaptability is one of the key motivations behind
OrthoSeg so that it serves as a useful architectural option for a wide range of
problems involving the task of semantic segmentation of coregistered multimodal
and/or multispectral imagery. Hence, OrthoSeg is designed to enable independent
scaling of parallel encoder networks and decoder network to better match
application requirements, such as the number of input channels, the effective
field-of-view, and model capacity.
