23 MAY 2014 • VOL 344 ISSUE 6186 789 SCIENCE sciencemag.org
guest-edited by Nosek and Lakens, is less
reassuring. All told, the researchers failed
to confirm the results of 10 well-known
studies, such as the social psychological effects of washing one’s hands, holding cups
of warm or cold liquid, or writing down
flattering things about oneself. In another
five cases, the replications found a smaller
effect than the original study did or encountered statistical complications it did
not report. For embodied cognition and
also for behavior priming—the study of
how exposure to one stimulus, such as the
word “dog,” changes one’s reaction to another, such as a photo of a cat—the results
are particularly grim. Seven of the replications focused on experiments in these areas, and all but one failed.

No one is suggesting misconduct in any
of the original studies, but the results are
further blows to a field shaken several years
ago when a towering figure in priming research, Diederik Stapel, confessed to faking
data (Science, 7 December 2012, p. 1270).
And earlier this month, Jens Förster of
the University of Amsterdam, a pioneer of
embodied cognition research, was accused
by a Dutch government-appointed ethics
panel of data manipulation—charges he denies (Science, 9 May, p. 566).

Nor should the results be taken as a general indictment of psychological research,
because the targeted studies were not a
random sample, Nosek says. “They are entirely cherry-picked,” he says, based on the
importance of the original study and the
feasibility of replicating it.

Some of the authors of the targeted studies, however, feel not just singled out but
persecuted. Schnall, for example, contends
that the replications were not held to the
same peer-review standard as her original
studies. “I stand by my methods and my
findings and have nothing to hide,” she says.

The replications did employ an alternative model of peer review, called preregistration, promoted by the Center for
Open Science, a nonprofit organization co-founded by Nosek (Science, 30 March 2012,
p. 1558). Before any data were collected, the
replicators submitted their experimental
design and data analysis plan to external
peer reviewers, including the principal investigator of the original study. The subsequent data analysis and conclusions were
reviewed only by Nosek or Lakens.

Schnall contends that Donnellan’s effortwas flawed by a “ceiling effect” that, es-sentially, discounted subjects’ most severemoral sentiments. “We tried a number ofstrategies to deal with her ceiling effectconcern,” Donnellan counters, “but it didnot change the conclusions.” Donnellanand his supporters say that Schnall simplytested too few people to avoid a false posi-tive result. (A colleague of Schnall’s, OliverGenschow, a psychologist at Ghent Univer-sity in Belgium, told Science in an e-mailthat he has successfully replicated Schnall’sstudy and plans to publish it.)Some replicators leaked news of theirfindings online, long before publicationand in dismissive terms. On his personalblog, Donnellan described his effort to re-peat Schnall’s research as an “epic fail” in aDecember post titled “Go Big or Go Home,”which was then widely circulated on Twit-ter. Donnellan defends the early announce-ment. “I feel badly, but the results are theresults,” he says.Schnall, however, says that her work was“defamed.” She believes she was denied alarge grant in part because of suspicionsabout her work and says that a reviewer ofone of her recently submitted papers “raisedthe issue about a ‘failed’ replication.” Sheadds that her graduate students “are wor-ried about publishing their work out of fearthat data detectives might come after themand try to find something wrong.”Other researchers whose work was tar-geted and failed to replicate told Sciencethat they have had experiences similar toSchnall’s. They all requested anonymity, forfear of what some in the field are calling“replication bullying.”Yet some whose findings did not hold upare putting a positive spin on the experi-ence. “This was certainly disappointing at apersonal level,” says Eugene Caruso, a psy-chologist at the University of Chicago BoothSchool of Business in Illinois, who in 2013reported a priming effect—exposing peopleto the sight of money made them more ac-cepting of societal norms—that failed toreplicate. “But when I take a broader per-spective, it’s apparent that we can alwayslearn something from a carefully designedand executed study.” Caruso now has alarger and more nuanced version of hisstudy under way.

The replications in psychology reflect agrowing trend in science (see table). Thefield’s bruising experience shows that suchefforts should be handled carefully, stressesDaniel Kahneman, a psychologist at Princ-eton University, whose work was success-fully replicated by the Many Labs team.“The relationship between authors andskeptics who doubt their findings is boundto be fraught,” he says. “It can be managedprofessionally if the rules that apply to bothsides are clearly laid out.”To reduce professional damage,Kahneman calls for a “replication eti-quette,” which he describes in a commen-tary published with the replications in

Social Psychology. For example, he says,“the original authors of papers should beactively involved in replication efforts”and “a demonstrable good-faith effort toachieve the collaboration of the originalauthors should be a requirement for pub-lishing replications.” In the case of thisweek’s replications, “the consultations didnot reach the level of author involvementthat I recommend.” However, he notes that“authors of low-powered studies with spec-tacular effects should not wait for hostilereplications: They should get in front of theproblem by replicating their own work.”For his part, Nosek hopes that the ten-sions will be short-lived growing painsas psychology adjusts to a demand, fromwithin and outside the field, for greater ac-countability. “Our primary aim is to makereplication entirely ordinary,” he says, “andmove it from a threat to a compliment.” ■Repeat after me

Select efforts in replication of research

EFFOR T REPLICATION TARGE T

Reproducibility Project: 50 high-impact cancer studies published from 2010 to 2012Cancer BiologyReproducibility Project: Articles published in 2008 from three psychology journalsPsychologyReproducibility Initiative Hub for authors to request independentreplications of their experimentsMany Labs project Global network for orchestrating large replicationsReproducibility in Checks software code in 613 applied computer science papersComputer ScienceCrowdstorming project More than 50 analysts address same researchquestion using shared data set