Under pressure from their employers and journals to produce statistically significant results, researchers turn to a number of tricks to make the numbers come out better.
It's a follow-up to their 2011 paper, "False Positive Psychology," which described some of the possible problems in psychology research. One of the major issues is "p-hacking," or the practice of changing assumptions or data in an experiment to make sure that the probability ("p") that an opposite hypothesis ("null") contradicts the research is below a certain level. Ultimately, "p-hacking" makes research less valid and increases the number of "false positives."
In response, the authors put together a 21-word statement every researcher should use as a disclosure, which they hope will make the field more transparent:
"We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study."
The authors perfectly sum up the transparency problem with an analogy and a photo. Whereas coffee shops are required to label milk containers, scientists don't have to "label their milk." In other words, researchers don't have to disclose what data they started out with, whether they took observations out, or whether they've dropped things from their model:
The table below of simulated results from their earlier paper shows how much these unreported techniques can impact statistical significance:
Joe Simmons, Leif Nelson and Uri Simonsohn
The lesson? Look for disclosures in any scientific paper, and always be skeptical.
Read the full article here
NOW READ: 18 Tips On Making Smarter Decisions