  Computer vision applications such as visual relationship detection and human
object interaction can be formulated as a composite (structured) set detection
problem in which both the parts (subject, object, and predicate) and the sum
(triplet as a whole) are to be detected in a hierarchical fashion. In this
paper, we present a new approach, denoted Part-and-Sum detection Transformer
(PST), to perform end-to-end visual composite set detection. Different from
existing Transformers in which queries are at a single level, we simultaneously
model the joint part and sum hypotheses/interactions with composite queries and
attention modules. We explicitly incorporate sum queries to enable better
modeling of the part-and-sum relations that are absent in the standard
Transformers. Our approach also uses novel tensor-based part queries and
vector-based sum queries, and models their joint interaction. We report
experiments on two vision tasks, visual relationship detection and human object
interaction and demonstrate that PST achieves state of the art results among
single-stage models, while nearly matching the results of custom designed
two-stage models.
