  Controlled experiments (A/B tests or randomized field experiments) are the de
facto standard to make data-driven decisions when implementing changes and
observing customer responses. The methodology to analyze such experiments
should be easily understandable to stakeholders like product and marketing
managers. Bayesian inference recently gained a lot of popularity and, in terms
of A/B testing, one key argument is the easy interpretability. For
stakeholders, "probability to be best" (with corresponding credible intervals)
provides a natural metric to make business decisions. In this paper, we
motivate the quintessential questions a business owner typically has and how to
answer them with a Bayesian approach. We present three experiment scenarios
that are common in our company, how they are modeled in a Bayesian fashion, and
how to use the models to draw business decisions. For each of the scenarios, we
present a real-world experiment, the results and the final business decisions
drawn.
