Last month, Richard Harris (of NPR fame) underlined the reproducibility crisis with an accessible narrative wittily yet aptly titled ‘Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hopes, and Wastes Billions’. In what has been described as a hard-hitting expose of the dysfunctional biomedical research system, Harris refers to a study led by Leonard P. Freedman, President of the Global Biological Standards Institute (GBSI), which found that approximately half of all preclinical research isn’t trustworthy.
And while many researchers may still be coming to terms with the crisis, this week Freedman and colleagues have delved even further into the issue at hand, publishing a comprehensive review of the initiatives being taken to improve reproducibility. Rather than focusing solely on the problem, ‘Reproducibility2020: Progress and priorities’ provides funders, publishers, researchers, and other stakeholders with actionable recommendations.
Much to our delight, the works of Harris and Freedman converge on our Preclinical Reproducibility and Robustness channel. This channel facilitates the open and transparent publication and discussion of confirmatory and non-confirmatory studies in biomedical research. Alongside our open data and method policies, this space was developed as part of our continued efforts to implement publishing practices which promote reproducibility.
Here, Leonard discusses the Reproducibility2020 initiative and offers some welcome optimism.
Why did GBSI decide to carry out this review?
We are optimistic for the future of research reproducibility, and this paper is really intended to demonstrate that the biomedical research community recognizes there is a problem and is committed to fixing it.
While much has been written about the various drivers that have contributed to irreproducibility in research, much less has been focused on the solutions. We undertook this evaluation because we believe that assessing and documenting the progress that has been made to date, and identifying the challenges that remain are equally important steps to resolution. The Report is the first comprehensive review of the initiatives being taken to improve reproducibility. It identifies action and impact that has been achieved by the biomedical research community and outlines priorities going forward.
This paper is really intended to demonstrate that the biomedical research community recognizes there is a problem and is committed to fixing it.
GBSI will update the Report annually, and new developments are posted frequently at GBSI.org. Hopefully, the lessons learned from these early efforts will assist all stakeholders seeking to scale up or replicate successful initiatives. For organizations exploring ways to become involved, the Report identifies key examples of the roles of funders, journals, researchers and other stakeholders, and provides recommended actions for future progress.
Can you give a brief overview of what the main sources of reproducibility are and how they can be addressed?
Since the widespread acknowledgement that reproducibility is a problem facing the biomedical research community, stakeholders have responded with innovation and policy change. The community is taking steps to work together and address the complexities of improving rigor and reproducibility, and the Report highlights many tangible examples of community-led actions.
The Report is the first comprehensive review of the initiatives being taken to improve reproducibility
We have grouped the causes of irreproducibility into the four general aspects that really make up the research process: study design and data analysis, reagents and reference materials, laboratory protocols, and reporting and review. But what sets the Report apart are the broad strategies for continued improvement of reproducibility, namely: 1) to drive research quality through strengthened journal and funder policies; 2) to create high quality online training and proficiency testing and make them accessible; 3) to engage all stakeholders to establish community-accepted standards and guidelines; and 4) to enhance open access to data and methodologies.
But the challenges are still great and some problems are more entrenched than others. Let’s start with incentive structure firmly in place in academia. This dynamic must seek a better balance between the pressures of career advancement and producing rigorous research. Ultimately, for impactful initiatives such as those described in the Report to be embraced by the community, the so-called perverse incentives currently driving academic success, such as publishing in the highest impact journals, or bringing in the biggest grant dollars, must be reassessed and ultimately changed.
What do you think has been the main changes in making biomedical research more reproducible since the ‘Case for Standards’ was first published?
By far the greatest progress over these few years has been in the stakeholders themselves recognizing the severity of the problem and the importance of taking active steps for improvement. Every stakeholder group is now addressing the issues, including journals, government funding agencies, private funders, academicians and industry. That’s crucial because there is not one simple fix—it is a community-wide problem and it will take a community-wide effort to achieve solutions.
By far the greatest progress over these few years has been in the stakeholders themselves recognizing the severity of the problem and the importance of taking active steps for improvement.
Can you tell us about the Reproducibility2020 Initiative?
GBSI launched the Reproducibility2020 Initiative in February 2016 as a challenge to all biomedical research community stakeholders to join with us to ensure that solutions are in place to improve reproducibility by the year 2020. As a leader in this global effort, GBSI is devoting its resources to affect change where its programs can make the greatest difference, work with partners to advance the broader agenda to improve the quality of scientific discovery in pre-clinical biological research settings, and keep stakeholders informed about movement and progress.
What plans for the future has GBSI to ensure reproducibility remains a priority in the future?
As a global leader in championing effective solutions to reproducibility, GBSI has recently completed a new 5-year strategic plan and will expand its work in the following areas:
Advancing standards and best practices to ensure quality and advance discovery in basic biomedical and translational research
- leading a global initiative toward improving the validation of reagents—particularly cells and antibodies
- working with community leaders to address the growing need for standards in emerging fields, such as regenerative medicine, engineering biology, and lab automation
Promoting education and training
- ensuring that high quality, accessible online training modules are available to both emerging and experienced researchers who are eager to improve their proficiencies in new and evolving best practices
- Advocating improved policies that increase rigor, accountability, and open access to data and methodologies by journals, funders, and academic institutions, and other research community stakeholders.
- In addition, GBSI will continue to bring the community together through a new membership program, expanded meetings and events, and publications of high impact for the field.