  Significant effort has been recently devoted to modeling visual relations.
This has mostly addressed the design of architectures, typically by adding
parameters and increasing model complexity. However, visual relation learning
is a long-tailed problem, due to the combinatorial nature of joint reasoning
about groups of objects. Increasing model complexity is, in general, ill-suited
for long-tailed problems due to their tendency to overfit. In this paper, we
explore an alternative hypothesis, denoted the Devil is in the Tails. Under
this hypothesis, better performance is achieved by keeping the model simple but
improving its ability to cope with long-tailed distributions. To test this
hypothesis, we devise a new approach for training visual relationships models,
which is inspired by state-of-the-art long-tailed recognition literature. This
is based on an iterative decoupled training scheme, denoted Decoupled Training
for Devil in the Tails (DT2). DT2 employs a novel sampling approach,
Alternating Class-Balanced Sampling (ACBS), to capture the interplay between
the long-tailed entity and predicate distributions of visual relations. Results
show that, with an extremely simple architecture, DT2-ACBS significantly
outperforms much more complex state-of-the-art methods on scene graph
generation tasks. This suggests that the development of sophisticated models
must be considered in tandem with the long-tailed nature of the problem.
