  Modern computer vision applications rely on learning-based perception modules
parameterized with neural networks for tasks like object detection. These
modules frequently have low expected error overall but high error on atypical
groups of data due to biases inherent in the training process. In building
autonomous vehicles (AV), this problem is an especially important challenge
because their perception modules are crucial to the overall system performance.
After identifying failures in AV, a human team will comb through the associated
data to group perception failures that share common causes. More data from
these groups is then collected and annotated before retraining the model to fix
the issue. In other words, error groups are found and addressed in hindsight.
Our main contribution is a pseudo-automatic method to discover such groups in
foresight by performing causal interventions on simulated scenes. To keep our
interventions on the data manifold, we utilize masked language models. We
verify that the prioritized groups found via intervention are challenging for
the object detector and show that retraining with data collected from these
groups helps inordinately compared to adding more IID data. We also plan to
release software to run interventions in simulated scenes, which we hope will
benefit the causality community.
