  Scientists often run experiments to distinguish competing theories. This
requires patience, rigor, and ingenuity - there is often a large space of
possible experiments one could run. But we need not comb this space by hand -
if we represent our theories as formal models and explicitly declare the space
of experiments, we can automate the search for good experiments, looking for
those with high expected information gain. Here, we present a general and
principled approach to experiment design based on probabilistic programming
languages (PPLs). PPLs offer a clean separation between declaring problems and
solving them, which means that the scientist can automate experiment design by
simply declaring her model and experiment spaces in the PPL without having to
worry about the details of calculating information gain. We demonstrate our
system in two case studies drawn from cognitive psychology, where we use it to
design optimal experiments in the domains of sequence prediction and
categorization. We find strong empirical validation that our automatically
designed experiments were indeed optimal. We conclude by discussing a number of
interesting questions for future research.
