  Continuous experimentation (CE) is used by many companies with
internet-facing products to improve their software based on user data. Some
companies deliberately adopt an experiment-driven approach to software
development while some companies use CE in a more ad-hoc fashion. The goal of
the study is to identify factors that explain the variations in the utility and
efficacy of CE between different companies. We conducted a multi-case study of
12 companies involved with CE and performed 27 interviewees with practitioners
at these companies. Based on that empirical data, we then built a theory of
factors at play in CE. We introduce a theory of Factors Affecting Continuous
Experimentation (FACE). The theory includes three factors, namely 1) processes
and infrastructure for CE, 2) the user problem complexity of the product
offering, and 3) incentive structures for CE. It explains how these factors
affect the effectiveness of CE and its ability to achieve problem-solution and
product-market fit. Our theory can be used by practitioners to assess an
organisation's potential for adopting CE, as well as, identifying factors which
pose challenges in gaining value from CE practices. Our results also provide a
starting point for further research on how contextual factors affect CE and how
these may be mitigated.
