  The problem of model selection with a limited number of experimental trials
has received considerable attention in cognitive science, where the role of
experiments is to discriminate between theories expressed as computational
models. Research on this subject has mostly been restricted to optimal
experiment design with analytically tractable models. However, cognitive models
of increasing complexity, with intractable likelihoods, are becoming more
commonplace. In this paper, we propose BOSMOS: an approach to experimental
design that can select between computational models without tractable
likelihoods. It does so in a data-efficient manner, by sequentially and
adaptively generating informative experiments. In contrast to previous
approaches, we introduce a novel simulator-based utility objective for design
selection, and a new approximation of the model likelihood for model selection.
In simulated experiments, we demonstrate that the proposed BOSMOS technique can
accurately select models in up to 2 orders of magnitude less time than existing
LFI alternatives for three cognitive science tasks: memory retention,
sequential signal detection and risky choice.
