  A/B experiments are commonly used in research to compare the effects of
changing one or more variables in two different experimental groups - a control
group and a treatment group. While the benefits of using A/B experiments are
widely known and accepted, there is less agreement on a principled approach to
creating software infrastructure systems to assist in rapidly conducting such
experiments. We propose a four-principle approach for developing a software
architecture to support A/B experiments that is domain agnostic and can help
alleviate some of the resource constraints currently needed to successfully
implement these experiments: the software architecture (i) must retain the
typical properties of A/B experiments, (ii) capture problem solving activities
and outcomes, (iii) allow researchers to understand the behavior and outcomes
of participants in the experiment, and (iv) must enable automated analysis. We
successfully developed a software system to encapsulate these principles and
implement it in a real-world A/B experiment.
