  Context: Continuous experimentation and A/B testing is an established
industry practice that has been researched for more than 10 years. Our aim is
to synthesize the conducted research.
  Objective: We wanted to find the core constituents of a framework for
continuous experimentation and the solutions that are applied within the field.
Finally, we were interested in the challenges and benefits reported of
continuous experimentation.
  Method: We applied forward snowballing on a known set of papers and
identified a total of 128 relevant papers. Based on this set of papers we
performed two qualitative narrative syntheses and a thematic synthesis to
answer the research questions.
  Results: The framework constituents for continuous experimentation include
experimentation processes as well as supportive technical and organizational
infrastructure. The solutions found in the literature were synthesized to nine
themes, e.g. experiment design, automated experiments, or metric specification.
Concerning the challenges of continuous experimentation, the analysis
identified cultural, organizational, business, technical, statistical, ethical,
and domain-specific challenges. Further, the study concludes that the benefits
of experimentation are mostly implicit in the studies.
  Conclusions: The research on continuous experimentation has yielded a large
body of knowledge on experimentation. The synthesis of published research
presented within include recommended infrastructure and experimentation process
models, guidelines to mitigate the identified challenges, and what problems the
various published solutions solve.
