  We propose Subject-Conditional Relation Detection SCoRD, where conditioned on
an input subject, the goal is to predict all its relations to other objects in
a scene along with their locations. Based on the Open Images dataset, we
propose a challenging OIv6-SCoRD benchmark such that the training and testing
splits have a distribution shift in terms of the occurrence statistics of
$\langle$subject, relation, object$\rangle$ triplets. To solve this problem, we
propose an auto-regressive model that given a subject, it predicts its
relations, objects, and object locations by casting this output as a sequence
of tokens. First, we show that previous scene-graph prediction methods fail to
produce as exhaustive an enumeration of relation-object pairs when conditioned
on a subject on this benchmark. Particularly, we obtain a recall@3 of 83.8% for
our relation-object predictions compared to the 49.75% obtained by a recent
scene graph detector. Then, we show improved generalization on both
relation-object and object-box predictions by leveraging during training
relation-object pairs obtained automatically from textual captions and for
which no object-box annotations are available. Particularly, for
$\langle$subject, relation, object$\rangle$ triplets for which no object
locations are available during training, we are able to obtain a recall@3 of
33.80% for relation-object pairs and 26.75% for their box locations.
