The key to the effectiveness of the Initiative is the collective action of reviewers. We suggest the following guidelines.
Reviewers
If you are a reviewer, check the author note of any paper you have accepted to review. If the authors have not certified compliance with the openness principles set forth in the Initiative — or given an account of why they cannot do so — then contact the action editor tell him or her that you would like the authors to certify that they meet the requirements:
I believe strongly in the value of openness and transparency. Please ask the authors on my behalf whether they can certify that they have met the standards of the Peer Reviewers’ Openness Initiative (https://opennessinitiative.org/).
If the authors cannot meet the basic requirements outlined in the Initiative, your review is very easy:
I cannot recommend this paper for publication, as it does not meet the minimum quality requirements for an open scientific manuscript (see https://opennessinitiative.org/). I would be happy to review a revision of the manuscript that corrects this critical oversight.
Do not outright decline to review a paper on the grounds that the paper does not meet the requirements of the Initiative. A minimal review, solely stating that the paper does not meet your minimum requirements for publication in the scientific literature, is much more valuable than declining to review because the authors will see your reasoning. Your responsibility as a peer-reviewer is quality control, and encouraging open data practices is part of that responsibility.
Finally, it is important to remember that the goal of the Initiative is to increase the quality of published research by creating the expectation of open research practices. It is therefore important not to use the Initiative in too legalistic a manner. We believe that researchers should be open with the products of their research in the vast majority of cases, particularly in the social and behavioral sciences; however, reasonable and clearly-explained exceptions can and should be made occasionally.
Other resources
- Joint Declaration of Data Citation Principles: The FORCE11 group’s statement of basic principles of data citation
- The Denton Declaration: An open data manifesto, which lays out broad principles of data management and sharing
- The Open Science Peer Review Oath: General guidelines for reviewers to follow as advocates of open science
- Uniform Principle for Sharing Integral Data and Materials Expeditiously (UPSIDE): Principles for the scientific community regarding sharing
- Transparency and Openness Promotion (TOP) Guidelines: Guidelines for journals regarding transparent research
- Alex Holcombe’s best practices for reviewers: a collaborative , work-in-progress document listing good reviewer practices in several categories (including data sharing)
- Further references
The guidelines indicate only what a reviewer ought to do in the event that an author(s) has not certified their manuscript fulfils the minimal quality requirements of the Initiative. What ought a reviewer do if the manuscript meets the minimal quality requirements of the Initiative? Is it appropriate for the reviewer to reject a manuscript if an alternative explanation of the data is possible?
I’m afraid that this is the way that the open data will be used, especially in light of the replication crisis plaguing social and experimental psychology at the moment. It does not seem at all appropriate for a reviewer to impede the publication of a manuscript because the reviewer finds an alternative explanation.
As the manuscript states, if the minimal requirements of the Initiative are met, then the review continues as it otherwise would have; nothing has changed. The Initiative says nothing about what is appropriate in review, otherwise. Typically, though, if a reviewer finds an alternative explanation of the data (which happens all the time, with or without open data) a reviewer would ask the authors to either mention it or provide evidence ruling it out. However, this is orthogonal to the Initiative.
I have tried this while reviewing for NeuroImage. Since there was no clarity about availability of data and code I asked the editor to request a clarification from the authors. The editor refused to do that explaining that this is not in compliance with the journal policy. The final decision was made without my feedback.