  Recent video question answering benchmarks indicate that state-of-the-art
models struggle to answer compositional questions. However, it remains unclear
which types of compositional reasoning cause models to mispredict. Furthermore,
it is difficult to discern whether models arrive at answers using compositional
reasoning or by leveraging data biases. In this paper, we develop a question
decomposition engine that programmatically deconstructs a compositional
question into a directed acyclic graph of sub-questions. The graph is designed
such that each parent question is a composition of its children. We present
AGQA-Decomp, a benchmark containing $2.3M$ question graphs, with an average of
$11.49$ sub-questions per graph, and $4.55M$ total new sub-questions. Using
question graphs, we evaluate three state-of-the-art models with a suite of
novel compositional consistency metrics. We find that models either cannot
reason correctly through most compositions or are reliant on incorrect
reasoning to reach answers, frequently contradicting themselves or achieving
high accuracies when failing at intermediate reasoning steps.
