Latest Issue of Science News

5/2/15 Cover

Scicurious

A peek behind the science curtain

Bethany Brookshire

Scicurious


Scicurious

A peer-reviewed study finds value in peer-reviewed research

A better funding score on grants is associated with more publications and patents

jury box

Scientists’ ideas are tried by a jury of their peers to determine if their grants get funded. A new study asks whether that jury is really identifying the best of the best.

Sponsor Message

The modern scientific process is guided and determined by peer review. A group of scientists in the field read and scrutinize a scientific manuscript before it is published. Committees of experts review plans for studies and determine whether they should be funded. The National Institutes of Health, National Science Foundation and the European Research Council distributed nearly $40 billion for scientific research in 2014, based almost entirely on peer review.

Do fellow scientists really pinpoint the best research plans to fund? Some scientists have worried that peer review pushes funding toward high profile institutions and previously successful researchers more than to potentially innovative research. A new study attempts to address this concern and finds that peer review does identify influential research. But evaluating whether peer review is the best way to identify the best science to fund is a much harder question to answer.

In the United States, some basic biomedical and clinical research is funded by private companies and investors. But much of it is funded by the NIH with government money. That raises the question of whether that money is being spent on research that is worthwhile for the population as a whole, whether it finds cures and uncovers the causes of disease. “Is the NIH still funding the path-breaking research that is likely to be influential?” asks Leila Agha, an economist at Boston University and a coauthor on the study. “In high-level applications, can [peer review] distinguish the best research?”

For all that science is a data-based enterprise, scientists don’t have the data on whether peer review really does end up funding the best research. “If you were a congressperson or a taxpayer, you might say ‘show me some data that peer review is good at picking things that turn out to be important,’” says Jeremy Berg, a biochemist at the University of Pittsburgh. “But until this study was done, the answer was ‘we believe it but we can’t prove it.’ As scientists, that’s kind of embarrassing.”

When a scientist wants to get NIH funding for a study, she writes up a grant proposal that reports results from preliminary studies, gives goals for the project, outlines the future experiments and estimates the time and resources they will require. The researcher submits her grant, and it’s assigned to a study section of 20 to 30 researchers who work in disciplines closely related to that of the grant proposal.

Within the study section, the grant will be assigned to three reviewers, two of whom provide detailed comments, and a reader, who provides additional comments. The reviewers will give the grant an overall score based on five criteria: Significance, scientific approach, potential innovation, the proposing scientist’s skills and whether the researcher’s university has the resources to support the work. About 40 to 50 percent of grants will be “triaged” at this stage. The rest go to the study section as a whole. After about 10 to 15 minutes of discussion, the grants receive final rankings by priority, with the lowest scores being the best and most likely to be funded.

In recent times, this means that most grants — even those that score well — will not get funding. NIH has a current annual budget of around $30 billion, but that number has not kept pace with the increasing number of scientists applying for research money. In 2014, only about 16 percent of new applications were funded. This makes applying for grants more competitive, and thus makes it even more important that peer review is selecting the research with the highest potential payoff.

Agha and Danielle Li, an economist at Harvard University, wanted to determine whether peer review could successfully predict the influence of the subsequent research. They examined the funding scores for a total of 137,215 peer-reviewed grants funded between 1980 and 2008. For each of the grants, they hunted down how many published scientific studies or patents the grant yielded within five years of the grant’s success. Li and Agha also looked at how many citations the scientific studies for each grant had accrued.

As they assessed the scores and the grants’ success rates, the researchers tried to factor out the scientists’ institutions, previous funding, previous work and field of study. The results showed that grants with higher scores did, in fact, tend to have more patents and more highly-cited publications associated with them. For each 10-point drop in score, a grant was 19 percent less likely to produce a high-impact publication and 14 percent less likely to produce a patent. The economists report their findings April 23 in Science.

“It’s good news. It’s suggesting that [grant reviewers] do on average have a clue,” says Lars Lefgren, an economist at Brigham Young University in Salt Lake City. “Some people complain that the NIH may be biased in terms of awarding grants to people with big names or established track records but who don’t have the most exciting or novel research. This study suggests those types of concerns may not happen on average.”

While the results do suggest that peer review can identify and fund impressive grant applications, Berg says that there are lots of hits and misses. Some very well-scored grants have relatively low publication records, and some grants that barely made the cut turned out to be superstars. “There’s a lot of scatter at every score,” he notes. But that’s to be expected. “It’s about predicting the future,” he explains. “A lot of scores are given for experiments that haven’t been performed yet. It’s like picking stocks: You do what you can do, but you don’t expect 100 percent accuracy.” And when money is tight and funding levels are extremely competitive, a grant proposal that just barely got funded is probably not actually a better proposal than the one that just barely missed the cut.

The study included a large percentage of grants that were competitive renewals. These are grants that have already been funded but are not complete and need a further five years of work. These grants tend to receive much higher scores and are much more likely to be funded than brand new research applications. “The more mature the research, the easier the task of figuring out if it will be successful,” says Agha. “But even with [new] grants, the relationship between peer review score and outcome is very much there.” Of course, many new grants have extensive preliminary data, often funded by previously successful grants, and so the previous success of the scientist is always going to come into play in scoring.

In addition, the study doesn’t show — and indeed can’t show — what path-breaking grant applications the peer review process may have missed. You can’t study the results of research that was never funded and potentially never performed. Berg says that one way to get at that question would be to calculate what might happen if the funding cutoff was more restrictive than it actually is. Researchers could then assess what grants would not have been funded and what research would have been lost.

And of course, Agha notes that their results can’t compare between the success rate of peer review as a system and other ways of choosing what research to fund, because there are no other systems available. “We’re not trying to make the case that the system is perfect or infallible,” she says.

But the study shows that peer review is adding something. Reviewers are not simply rewarding their high-profile colleagues and are choosing studies that are associated with future productivity. “Everyone realizes [peer review] has flaws and biases, but it’s relatively transparent and it has done a good job,” Berg notes. “There have been important discoveries,” It’s like what they say about democracy, he says: “It’s the worst system, except for all the others.” 

Neuroscience

Serotonin and the science of sex

By Bethany Brookshire 8:00am, April 10, 2015
Some scientists say that low serotonin makes male mice mate with males and females. Others disagree. In the end, it’s not about sexual preference, but about how science works.
Science & Society

Women in engineering engage best with gender parity

By Bethany Brookshire 5:16pm, April 6, 2015
There are many hypotheses as to why women don’t stay in science or engineering. A new study puts an intervention to the test.
Neuroscience

Our taste in music may age out of harmony

By Bethany Brookshire 11:44am, March 27, 2015
Age-related hearing loss may be more than just the highest notes. The brain may also lose the ability to tell consonance from dissonance, a new study shows.
Neuroscience

Sniffing out human pheromones

By Bethany Brookshire 12:00pm, March 12, 2015
A new review argues that most of the chemicals labeled human pheromones, and the experiments behind them, don’t pass the smell test.
Nutrition

Report offers stimulating recommendation on coffee

By Bethany Brookshire 3:13pm, March 4, 2015
Results from a committee of experts give the blessing to moderate coffee intake. But as we all raise our mugs, the science behind the report is worth a closer look.
Health

There’s more than one way to persuade people to vaccinate

By Bethany Brookshire 4:14pm, February 19, 2015
Fear, facts and attitude are all strategies for promoting immunization
Psychology,, Science & Society

Scientists of a feather flock together

By Bethany Brookshire 8:00am, February 12, 2015
When it comes to major scientific issues such as global warming and GMOs, scientists and the public don’t see eye to eye. It might be because socially, they don’t see each other at all.
Neuroscience

How the brain sees follow-through

By Bethany Brookshire 4:29pm, February 4, 2015
The follow-through on your golf swing is more than just a way to use up extra energy. It’s part of how your brain “sees” a movement.
Science & Society,, Psychology

Attitude, not aptitude, may contribute to the gender gap

By Bethany Brookshire 5:26pm, January 15, 2015
Does talent or hard work matter most? A new survey suggests an emphasis on genius predicts how many women end up in a field of study.
Neuroscience,, Mental Health

To beat sleepiness of anxiety drugs, team looks to body’s clock

By Bethany Brookshire 3:26pm, January 14, 2015
Studying basic functions, such as the body’s clock, has inadvertently led to a compound that relieves anxiety in mice.
Subscribe to RSS - Scicurious