  Online labor markets have great potential as platforms for conducting
experiments, as they provide immediate access to a large and diverse subject
pool and allow researchers to conduct randomized controlled trials. We argue
that online experiments can be just as valid---both internally and
externally---as laboratory and field experiments, while requiring far less
money and time to design and to conduct. In this paper, we first describe the
benefits of conducting experiments in online labor markets; we then use one
such market to replicate three classic experiments and confirm their results.
We confirm that subjects (1) reverse decisions in response to how a
decision-problem is framed, (2) have pro-social preferences (value payoffs to
others positively), and (3) respond to priming by altering their choices. We
also conduct a labor supply field experiment in which we confirm that workers
have upward sloping labor supply curves. In addition to reporting these
results, we discuss the unique threats to validity in an online setting and
propose methods for coping with these threats. We also discuss the external
validity of results from online domains and explain why online results can have
external validity equal to or even better than that of traditional methods,
depending on the research question. We conclude with our views on the potential
role that online experiments can play within the social sciences, and then
recommend software development priorities and best practices.
