Science & the Public | Science News

SUPPORT SCIENCE NEWS

Science News is a nonprofit.

Help us keep you informed.

Science & the Public

Where science
and society meet

Science News

Science & the Public


Science & the Public

‘Replication crisis’ spurs reforms in how science studies are done

But some researchers say the focus on reproducibility ignores a larger problem

The Thinker statue

PONDERING SCIENCE  A new investigation indicates that reproducibility of social science studies (including one investigating how contemplation of Rodin’s The Thinker statue affects religious beliefs), while not great, is improving. But intensifying replication efforts may address only surface problems.

Sponsor Message

What started out a few years ago as a crisis of confidence in scientific results has evolved into an opportunity for improvement. Researchers and journal editors are exposing how studies get done and encouraging independent redos of published reports. And there’s nothing like the string of failed replications to spur improved scientific practice.

That’s the conclusion of a research team, led by Caltech economist Colin Camerer, that examined 21 social science papers published in two major scientific journals, Nature and Science, from 2010 to 2015. Five replication teams directed by coauthors of the new study successfully reproduced effects reported for 13 of those investigations, the researchers report online August 27 in Nature Human Behavior. Results reported in eight papers could not be replicated.

The new study is an improvement over a previous attempt to replicate psychology findings (SN: 4/2/16, p. 8). But the latest results underscore the need to view any single study with caution, a lesson that many researchers and journal gatekeepers have taken to heart over the past few years, Camerer’s team says. An opportunity now exists to create a scientific culture of replication that provides a check on what ends up getting published and publicized, the researchers contend.

Still, the new study reveals a troubling aspect of successful experimental redos. Camerer’s team found that for repeat studies that panned out, which included four to five times as many participants as originally studied, the statistical strength to detect actual effects was weaker than reported for the initial investigations. In other words, the best replications — which exceeded initial studies in their ability to detect actual effects — were only partially successful.

One reason for that trend is that scientific journals have tended not to publish studies that disconfirm previous findings, leaving initial findings unchallenged until now, says study coauthor and psychologist Brian Nosek of the University of Virginia in Charlottesville. Even the most prestigious journals have often published results that garner lots of scientific and media attention but that could easily have occurred randomly, he says.

On the plus side, the new report appears as such practices are changing. “The social and behavioral sciences are in the midst of a reformation in scientific practices,” Nosek says.

In the last five years, for example, 19 of 33 journals in social and personality psychology have established policies requiring investigators to submit their research designs for peer review before submitting research papers for review. In this way, peer reviewers can check whether experimenters altered their procedures to tease out positive effects. The same journals also collect experimental data from researchers so that replications can be conducted.

Intriguingly, when Camerer’s group asked a group of nearly 400 researchers, mostly psychologists and economists, to examine data from the 21 experiments and predict whether each could be reproduced, the scientists’ forecasts were usually correct. Peer predictions may be one way to bolster peer reviews and help weed out weak studies, Nosek says.

Another positive sign is that scientists whose papers were rechecked in the new study generally cooperated with the effort, even if their findings failed to replicate. For instance, one new replication study that Camerer and colleagues examined did not support a 2012 Science report that viewing pictures of Auguste Rodin’s famous statue The Thinker reduces volunteers’ self-reported religious belief. This finding was part of a project examining how mental reflection affects religious belief.

Psychologist Will Gervais of the University of Kentucky in Lexington, a coauthor of the 2012 paper, welcomes the new evidence. “In hindsight, we oversold a study with an effect that was barely statistically significant,” Gervais says. “I like to think that our study wouldn’t get published today.”

Current replication efforts represent “an opportunity to sharpen our scientific practices,” he adds.

Even if he’s right, the problem with studies in the social sciences, as well as disciplines including neuroscience and medical research (SN: 2/18/17, p. 10), goes deeper than reproducibility, says psychologist Gerd Gigerenzer of the Max Planck Institute for Human Development in Berlin.

Researchers in these fields largely rely on a statistical technique that assesses whether a result — say, an apparent decline in religious belief after viewing The Thinker — would likely occur if there were no true difference. Scientists call that a null hypothesis. An arbitrary cutoff is used to determine whether a reported difference from a null hypothesis is “statistically significant.” But that doesn’t establish the existence of a true effect, although it’s usually assumed to do just that. And typically, researchers make no attempt to test alternative predictions that attempt to explain how, for example, contemplation might affect spiritual convictions, Gigerenzer says.

What’s worse, many researchers wrongly assume that achieving statistical significance in a study makes replication of its results unnecessary.

That’s what Gigerenzer found in a review of studies that quizzed 839 academic psychologists and 991 undergraduate and graduate psychology students. Of those, 20 percent of faculty teaching statistics, 39 percent of professors and lecturers, and 66 percent of students believed that statistical significance means a result requires no replication, Gigerenzer reports in the June Advances in Methods and Practices in Psychological Science.

Such results fit a scenario in which researchers usually resort to null-hypothesis testing as a ritual for getting studies published, without having to develop actual theories, Gigerenzer contends. Psychology departments need to teach students about various statistical methods that can be used to test experimental predictions, he suggests. Journal editors should no longer accept studies based solely on statistical significance, he adds.

“We need to blow up the current system and promote real statistical thinking and judgment,” Gigerenzer argues.

Although not a proponent of radical change, Nosek sees bigger revisions on the horizon. “I’m going to be optimistic and predict that in five years we’ll see the social and behavioral sciences move beyond studying null hypotheses,” he says.

Genetics,, Science & Society

Most Americans think it’s OK to tweak a baby’s genes to prevent disease

By Tina Hesman Saey 6:26pm, July 26, 2018
Americans generally favor tweaking a baby’s genes to reduce the chance of getting a disease, but think boosting intelligence is a step too far.
Genetics,, Science & Society

Privacy and consumer genetic testing don’t always mix

By Cassie Martin 7:00am, June 5, 2018
Interested in taking a direct-to-consumer genetic test? Here are some things you should know.
Earth

No, Kilauea won’t cause mass destruction

By Carolyn Gramling 5:02pm, May 16, 2018
A steam explosion at Kilauea isn’t anything like the explosive eruptions of certain other volcanoes.
Science & Society

An antiscience political climate is driving scientists to run for office

By Rachel Ehrenberg 4:03pm, April 13, 2018
Hoping to inject evidence-based science into policy, more scientists are putting their name on the ballot.
Science & Society,, Climate

What we can and can’t say about Arctic warming and U.S. winters

By Carolyn Gramling 9:00am, March 16, 2018
Evidence of a connection is growing stronger, but scientists still struggle to explain why.
Numbers

Forget Pi Day. We should be celebrating Tau Day

By Emily Conover 11:30am, March 14, 2018
Pi Day may be fun, but it’s based on a flawed mathematical constant.
Computing,, Science & Society

Are computers better than people at predicting who will commit another crime?

By Maria Temming 9:00am, February 20, 2018
If crime-predicting computer programs aren’t any more accurate than human guesswork, do they still have a place in the criminal justice system?
Science & Society

4 questions about the new U.S. budget deal and science

By Emily DeMarco 6:16pm, February 9, 2018
A new spending package could lead to U.S. science agencies getting a bump in funding.
Science & Society

Revisiting the science stories that made us cry, think and say ‘OMG’ in 2017

By Kate Travis 12:00pm, December 27, 2017
Each year Science News selects the top stories for their importance and impact. But the staff’s favorite stories strike a different chord.
Science & Society,, Genetics

Parents may one day be morally obligated to edit their baby’s genes

By Tina Hesman Saey 7:00am, November 28, 2017
The CRISPR debate is moving from “should we or shouldn’t we?” to “do we have to?”
Subscribe to RSS - Science & the Public