  When interpreting A/B tests, we typically focus only on the statistically
significant results and take them by face value. This practice, termed
post-selection inference in the statistical literature, may negatively affect
both point estimation and uncertainty quantification, and therefore hinder
trustworthy decision making in A/B testing. To address this issue, in this
paper we explore two seemingly unrelated paths, one based on supervised machine
learning and the other on empirical Bayes, and propose post-selection
inferential approaches that combine the strengths of both. Through large-scale
simulated and empirical examples, we demonstrate that our proposed
methodologies stand out among other existing ones in both reducing
post-selection biases and improving confidence interval coverage rates, and
discuss how they can be conveniently adjusted to real-life scenarios.
