Researchers have known for years just how sketchy preclinical biomedical research can be. Reports on research projects that can capture headlines around the world are also not infrequently impossible to reproduce. And now a new study has attempted to put a dollar figure on the amount of research produced in the U.S. each year that can't be reproduced.
Their bottom line: $28 billion, or slightly more than half of all the work, is spent on work that cannot be reproduced for a variety of reasons, according to a report from Nature.
There are some big caveats to be made. The study authors--Len Freedman, head of the Global Biological Standards Institute in Washington, was the lead author, working with Boston University economist Iain Cockburn and others--note that they're making a broad ballpark estimate based on a few likely culprits.
"Clearly, there are tremendous inefficiences [in research], and this is putting a spotlight on that," Freedman tells Nature.
Their estimate was based on the belief that materials contributed to a 36% failure rate, with study design coming in at 28% and data analysis at 26%.
The high failure rate will come as no surprise to anyone in the field. Earlier reports suggested that pharma researchers are often frustrated when it comes to reproducing the studies that come to their attention. Failure rates as high as 75% have been recorded, according to Nature, making the new study's estimate of a 53% irreproducible rate rather conservative. Critics of the way preclinical research is done have also spotlighted how journals' tendency to prefer promising studies over setbacks has also tended to skew outcomes.
The investigators in this study, though, aren't trying to draw more attention for the way the system often breaks down now. Rather, they want to see some best standards adopted to cut down on the waste and make the field more productive.
"The message is less about 'Oh my god, we're flushing $20 billion down the toilet'," Freedman tells Nature, "and more about 'Here is an opportunity to increase efficiencies to get more bang for the buck'."
- here's the story from Nature