Should the NIH scrap its current peer-review model to determine which studies it should bankroll and turn to a more random lottery system? A trio of researchers says yes, pointing to a new study of their own that shows just how hard it can be to judge the best scientific projects among top-tier proposals.
Under the current system, the NIH uses a peer-reviewed approach to try and spotlight the very best science as it looks to back the top 10%, explains a report from Dr. Ferric Fang at the University of Washington, Anthony Bowen at the Albert Einstein College of Medicine and Dr. Arturo Casadevall at the Johns Hopkins Bloomberg School of Public Health.
Due to congressional stinginess, the NIH's budget of $30 billion leaves a vast number of proposals on the shelf, and getting a piece of that money has become an increasingly controversial subject in research circles.
The problem with the current system, according to the authors, is that it really does a poor job in distinguishing which of the top projects are the best. It actually only works in shoving aside the bottom half of the proposals, relegating the worst science to the sidelines. And when a single negative remark can scuttle any chance of success, they add, the NIH--which spends $110 million a year on peer review--may wind up ignoring the kind of work that promises the best payback.
"When people's opinions count a lot, we may be doing worse than choosing at random," Casadevall says. "A negative word at the table can often swing the debate. And this is how we allocate research funding in this country."
The investigators reached their conclusion after re-examining data on 102,740 studies funded by the NIH over 28 years through 2008. By focusing only on the top 20%, they found that the system was no better at identifying studies in the 20th percentile group than the top 10%, meaning that a random lottery system restricted to the top tier would be just as likely to work as trying to ID the 10% which are supported these days. And that would be considerably less expensive as well as less controversial.
- here's the release