  Machine learning algorithms can find predictive signals that researchers fail
to notice; yet they are notoriously hard-to-interpret. How can we extract
theoretical insights from these black boxes? History provides a clue. Facing a
similar problem -- how to extract theoretical insights from their intuitions --
researchers often turned to ``anomalies:'' constructed examples that highlight
flaws in an existing theory and spur the development of new ones. Canonical
examples include the Allais paradox and the Kahneman-Tversky choice experiments
for expected utility theory. We suggest anomalies can extract theoretical
insights from black box predictive algorithms. We develop procedures to
automatically generate anomalies for an existing theory when given a predictive
algorithm. We cast anomaly generation as an adversarial game between a theory
and a falsifier, the solutions to which are anomalies: instances where the
black box algorithm predicts - were we to collect data - we would likely
observe violations of the theory. As an illustration, we generate anomalies for
expected utility theory using a large, publicly available dataset on real
lottery choices. Based on an estimated neural network that predicts lottery
choices, our procedures recover known anomalies and discover new ones for
expected utility theory. In incentivized experiments, subjects violate expected
utility theory on these algorithmically generated anomalies; moreover, the
violation rates are similar to observed rates for the Allais paradox and Common
ratio effect.
