  Video relation detection forms a new and challenging problem in computer
vision, where subjects and objects need to be localized spatio-temporally and a
predicate label needs to be assigned if and only if there is an interaction
between the two. Despite recent progress in video relation detection, overall
performance is still marginal and it remains unclear what the key factors are
towards solving the problem. Following examples set in the object detection and
action localization literature, we perform a deep dive into the error diagnosis
of current video relation detection approaches. We introduce a diagnostic tool
for analyzing the sources of detection errors. Our tool evaluates and compares
current approaches beyond the single scalar metric of mean Average Precision by
defining different error types specific to video relation detection, used for
false positive analyses. Moreover, we examine different factors of influence on
the performance in a false negative analysis, including relation length, number
of subject/object/predicate instances, and subject/object size. Finally, we
present the effect on video relation performance when considering an oracle fix
for each error type. On two video relation benchmarks, we show where current
approaches excel and fall short, allowing us to pinpoint the most important
future directions in the field. The tool is available at
\url{https://github.com/shanshuo/DiagnoseVRD}.
