  Reference Expression Segmentation (RES) and Reference Expression Generation
(REG) are mutually inverse tasks that can be naturally jointly trained. Though
recent work has explored such joint training, the mechanism of how RES and REG
can benefit each other is still unclear. In this paper, we propose a unified
mutual supervision framework that enables two tasks to improve each other. Our
mutual supervision contains two directions. On the one hand, Disambiguation
Supervision leverages the expression unambiguity measurement provided by RES to
enhance the language generation of REG. On the other hand, Generation
Supervision uses expressions automatically generated by REG to scale up the
training of RES. Such unified mutual supervision effectively improves two tasks
by solving their bottleneck problems. Extensive experiments show that our
approach significantly outperforms all existing methods on REG and RES tasks
under the same setting, and detailed ablation studies demonstrate the
effectiveness of all components in our framework.
