  We propose a fully differentiable architecture for simultaneous semantic and
instance segmentation (a.k.a. panoptic segmentation) consisting of a
convolutional neural network and an asymmetric multiway cut problem solver. The
latter solves a combinatorial optimization problem that elegantly incorporates
semantic and boundary predictions to produce a panoptic labeling. Our
formulation allows to directly maximize a smooth surrogate of the panoptic
quality metric by backpropagating the gradient through the optimization
problem. Experimental evaluation shows improvement by backpropagating through
the optimization problem w.r.t. comparable approaches on Cityscapes and COCO
datasets. Overall, our approach shows the utility of using combinatorial
optimization in tandem with deep learning in a challenging large scale
real-world problem and showcases benefits and insights into training such an
architecture.
