Over 100 major psychology findings were retested by a team of independent researchers, and they could only reproduce 36 percent of the original findings.
Science hinges on the idea that research findings are reliable because they can be tested over and over again, producing the same results. But the social sciences have raised some red flags concerning the reliability of study results. A team of researchers tried to reproduce the results of 100 psychology studies, but less than half of the attempts yielded the same results as the first studies.
In fact, the researchers found problems with over 60 of the studies. One study touched upon free will, stating that participants who read a passage supporting the idea that behavior is predetermined were more likely to cheat on a subsequent test than those who hadn’t read the passage. Another, on mate preference, found that taken women were more likely to rate the attractiveness of single men highly when the women were highly fertile. These studies, along with dozens of others, simply produced weaker effects when they were retested.
The initiative, called Reproducibility Project: Psychology, was taken on by an international team of 270 experts who repeated 100 experiments on five different continents. It was initially launched in the US in response to rising concerns about the reliability of psychology research. Brian Nosek, a psychology professor who led the study at the University of Virginia, told The Guardian that he’s disappointed that the study didn’t find the effects of psychology studies to be more reproducible, and that he thinks psychologists can do better.
“The key caution that an average reader should take away is any one study is not going to be the last word,” Nosek added. “Science is a process of uncertainty reduction, and no one study is almost ever a definitive result on its own.”
Perhaps psychologists have a difficult field to work with because they rely heavily on self-reported data. They also explore topics that are highly subjective and influenced by a number of biases— difference of opinions, gender discrepancies, cultural factors, and more.
However, Dr. John Ioannidis, a director of Stanford University’s Meta-Research Innovation Center, speculates that the problem could be even worse in other fields, including cell biology, economics, neuroscience, clinical medicine, and animal research. Unfortunately, it would be much more costly to re-examine the results of experiments in medicine, cancer biology, and brain science.
There were three main reasons that the retested studies didn’t hold up to review. Most replication teams contacted original study authors to make sure they used the same materials and methods, but small differences, like when and how the study was reproduced, might have influenced the results. For example, Paola Bressan, a psychologist at the University of Padua and one of the original authors of the mate preference study, noted that her sample of women were mostly Italian as opposed to American psychology students.
The replication studies could also have simply failed to detect the original result (a “false negative”). Conversely, the original result could have been a false positive. But no matter the reason for the discrepancies in the results, some psychologists are welcoming the scrutiny of their work.
Alan Kraut, the executive director of the Association for Psychological Science, New York Times, “It’s like we’ve come clean. This kind of correction is something that has to happen across science, and I’m proud that psychology is leading the charge on this.”
Many other psychologists agree that the results brought an extremely important issue to the forefront. This attitude indicates a wider cultural shift within the field of psychology. Just five years ago, psychology researchers acted as their own editors, crafting the story that would be told by their data. Even before the publication of this new report, researchers around the world had started encouraging transparency and data sharing.
Jelte Wicherts, an associate professor in the department of statistics and methods at the University of Tilburg in the Netherlands, told NYT, “We knew there were many results that were too good to be true. It’s interesting. I’ve just joined a faculty where young researchers, they’ve completely changed their ways. They share all their data on request, without any regulations; they put everything online before sending out papers for review. It’s a grass-roots effort.”
While the field of psychology is certainly in for a big transformation, this study illuminates the importance of staying skeptical. When it comes down to it, all studies have certain limitations that must be considered before jumping to the conclusion that scientific results are 100 percent accurate. Plus, it’s safe to assume that these findings will drive scientists to ensure their experimental conclusions are as reliable as possible. There’s always room for improvement, and even science isn’t immune to that fact.
Facebook comments