  The recently introduced panoptic segmentation task has renewed our
community's interest in unifying the tasks of instance segmentation (for thing
classes) and semantic segmentation (for stuff classes). However, current
state-of-the-art methods for this joint task use separate and dissimilar
networks for instance and semantic segmentation, without performing any shared
computation. In this work, we aim to unify these methods at the architectural
level, designing a single network for both tasks. Our approach is to endow Mask
R-CNN, a popular instance segmentation method, with a semantic segmentation
branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly,
this simple baseline not only remains effective for instance segmentation, but
also yields a lightweight, top-performing method for semantic segmentation. In
this work, we perform a detailed study of this minimally extended version of
Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust
and accurate baseline for both tasks. Given its effectiveness and conceptual
simplicity, we hope our method can serve as a strong baseline and aid future
research in panoptic segmentation.
