  We present a novel Tensor Composition Net (TCN) to predict visual
relationships in images. Visual Relationship Prediction (VRP) provides a more
challenging test of image understanding than conventional image tagging and is
difficult to learn due to a large label-space and incomplete annotation. The
key idea of our TCN is to exploit the low-rank property of the visual
relationship tensor, so as to leverage correlations within and across objects
and relations and make a structured prediction of all visual relationships in
an image. To show the effectiveness of our model, we first empirically compare
our model with Multi-Label Image Classification (MLIC) methods, eXtreme
Multi-label Classification (XMC) methods, and VRD methods. We then show that
thanks to our tensor (de)composition layer, our model can predict visual
relationships which have not been seen in the training dataset. We finally show
our TCN's image-level visual relationship prediction provides a simple and
efficient mechanism for relation-based image-retrieval even compared with VRD
methods.
