  Reference Expression Generation (REG) and Comprehension (REC) are two highly
correlated tasks. Modeling REG and REC simultaneously for utilizing the
relation between them is a promising way to improve both. However, the problem
of distinct inputs, as well as building connections between them in a single
model, brings challenges to the design and training of the joint model. To
address the problems, we propose a unified model for REG and REC, named UniRef.
It unifies these two tasks with the carefully-designed Image-Region-Text Fusion
layer (IRTF), which fuses the image, region and text via the image
cross-attention and region cross-attention. Additionally, IRTF could generate
pseudo input regions for the REC task to enable a uniform way for sharing the
identical representation space across the REC and REG. We further propose
Vision-conditioned Masked Language Modeling (VMLM) and Text-Conditioned Region
Prediction (TRP) to pre-train UniRef model on multi-granular corpora. The VMLM
and TRP are directly related to REG and REC, respectively, but could help each
other. We conduct extensive experiments on three benchmark datasets, RefCOCO,
RefCOCO+ and RefCOCOg. Experimental results show that our model outperforms
previous state-of-the-art methods on both REG and REC.
