  A significant bottleneck in training deep networks for part segmentation is
the cost of obtaining detailed annotations. We propose a framework to exploit
coarse labels such as figure-ground masks and keypoint locations that are
readily available for some categories to improve part segmentation models. A
key challenge is that these annotations were collected for different tasks and
with different labeling styles and cannot be readily mapped to the part labels.
To this end, we propose to jointly learn the dependencies between labeling
styles and the part segmentation model, allowing us to utilize supervision from
diverse labels. To evaluate our approach we develop a benchmark on the
Caltech-UCSD birds and OID Aircraft dataset. Our approach outperforms baselines
based on multi-task learning, semi-supervised learning, and competitive methods
relying on loss functions manually designed to exploit sparse-supervision.
