  In this paper, we draw attention to a problem that is often overlooked or
ignored by companies practicing hypothesis testing (A/B testing) in online
environments. We show that conducting experiments on limited inventory that is
shared between variants in the experiment can lead to high false positive rates
since the core assumption of independence between the groups is violated. We
provide a detailed analysis of the problem in a simplified setting whose
parameters are informed by realistic scenarios. The setting we consider is a
$2$-dimensional random walk in a semi-infinite strip. It is rich enough to take
a finite inventory into account, but is at the same time simple enough to allow
for a closed form of the false-positive probability. We prove that high
false-positive rates can occur, and develop tools that are suitable to help
design adequate tests in follow-up work. Our results also show that high
false-negative rates may occur. The proofs rely on a functional limit theorem
for the $2$-dimensional random walk in a semi-infinite strip.
