  Two-sided marketplaces are standard business models of many online platforms
(e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers,
buyers or content viewers on one side and producers, sellers or
content-creators on the other. Consumer side measurement of the impact of a
treatment variant can be done via simple online A/B testing. Producer side
measurement is more challenging because the producer experience depends on the
treatment assignment of the consumers. Existing approaches for producer side
measurement are either based on graph cluster-based randomization or on certain
treatment propagation assumptions. The former approach results in low-powered
experiments as the producer-consumer network density increases and the latter
approach lacks a strict notion of error control. In this paper, we propose (i)
a quantification of the quality of a producer side experiment design, and (ii)
a new experiment design mechanism that generates high-quality experiments based
on this quantification. Our approach, called UniCoRn (Unifying Counterfactual
Rankings), provides explicit control over the quality of the experiment and its
computation cost. Further, we prove that our experiment design is optimal to
the proposed design quality measure. Our approach is agnostic to the density of
the producer-consumer network and does not rely on any treatment propagation
assumption. Moreover, unlike the existing approaches, we do not need to know
the underlying network in advance, making this widely applicable to the
industrial setting where the underlying network is unknown and challenging to
predict a priori due to its dynamic nature. We use simulations to validate our
approach and compare it against existing methods. We also deployed UniCoRn in
an edge recommendation application that serves tens of millions of members and
billions of edge recommendations daily.
