  Characterizing the patterns of errors that a system makes helps researchers
focus future development on increasing its accuracy and robustness. We propose
a novel form of "meta learning" that automatically learns interpretable rules
that characterize the types of errors that a system makes, and demonstrate
these rules' ability to help understand and improve two NLP systems. Our
approach works by collecting error cases on validation data, extracting
meta-features describing these samples, and finally learning rules that
characterize errors using these features. We apply our approach to VilBERT, for
Visual Question Answering, and RoBERTa, for Common Sense Question Answering.
Our system learns interpretable rules that provide insights into systemic
errors these systems make on the given tasks. Using these insights, we are also
able to "close the loop" and modestly improve performance of these systems.
