  We consider the design of experiments to evaluate treatments that are
administered by self-interested agents, each seeking to achieve the highest
evaluation and win the experiment. For example, in an advertising experiment, a
company wishes to evaluate two marketing agents in terms of their efficacy in
viral marketing, and assign a contract to the winner agent. Contrary to
traditional experimental design, this problem has two new implications. First,
the experiment induces a game among agents, where each agent can select from
multiple versions of the treatment it administers. Second, the action of one
agent -- selection of treatment version -- may affect the actions of another
agent, with the resulting strategic interference complicating the evaluation of
agents. An incentive-compatible experiment design is one with an equilibrium
where each agent selects its natural action, which is the action that would
maximize the performance of the agent if there was no competition (e.g.,
expected number of conversions if agent was assigned the contract). Under a
general formulation of experimental design, we identify sufficient conditions
that guarantee incentive-compatible experiments. These conditions rely on the
existence of statistics that can estimate how agents would perform without
competition, and their use in constructing score functions to evaluate the
agents. In the setting with no strategic interference, we also study the power
of the design, i.e., the probability that the best agent wins, and show how to
improve the power of incentive-compatible designs. From the technical side, our
theory uses a range of statistical methods such as hypothesis testing,
variance-stabilizing transformations and the Delta method, all of which rely on
asymptotics.
