18 thoughts on “Signatories”

1. Congratulations on a really nice initiative, great agenda and paper. However I am surprised to find an important omission. I think it is essential that the main piece of data, the paper manuscript itself, must also be made publicly freely available, i.e. it has to be published open-access. Perhaps it is somehow implicit in the open data policies described in the agenda, but I think this is important enough to deserve a point by itself.

   Would you be able to add that point to the agenda?

   Reply

   1. Hi Jose, while we agree in principle that open access is important, the institutional barriers that prevent open access are different than those preventing open data. It would thus require a different sort of initiative than the one here, which emphasizes open data in review. Reviewers have no real power over whether the authors can pay open access fees, or what journal the authors submit to.

      Reply

      1. I would still insist that an open data policy must come together with an open-access policy. In the end of the day the paper is just another piece of data, actually the most human-readable part of the data. There’s not much sense in making lots of data openly available while the paper that explains it remains hidden behind a paywall.

         Regarding reviewers’ power on deciding what journal the authors submits to: isn’t the point of this agenda precisely the reviewer’s power to push authors to release the data openly? I think reviewers could also be equally effective in pushing the authors to decide for an open access journal. The fees can surely not be a deciding factor when journals like peerj.com offer extremely low fees.

         Don’t get me wrong, I completely agree with you guys: open data is extremely important. But open access is as essential and should come as a necessary prerequisite.

         Reply

2. Hi Jose, PeerJ looks really interesting, thanks for pointing it out. In psychology at least, there is a lot of institutional pressure to publish via standard routes, which happen unfortunately to not be Open Access Gold. I agree with Richard that reviewing for paid-access journals is not an effective lever for encouraging people to publish in open-access gold journals 🙂 But there is also Open Access Green. Most paid-access journals allow publication of a substantively-final version of the paper in your institutional repository and search engines like BASE allow search across repositories. So one could easily encourage open access green through the review process of paid-access journals. I’m agnostic as to whether this should be retrofitted to this agenda of the subject of another other.

   Reply

   1. Thanks for bringing up green openaccess, that Richard neglected to mention. Some publishers don’t allow a substantively-final (author-generated, not the publisher PDF) version to be uploaded to repositories, but even those allow a preprint (before peer review) to be uploaded to repositories.

      About similar initiatives in that area, some of us have been selectively reviewing for open access journals for several years. Eighty-four are listed at this website http://www.openaccesspledge.com/?page_id=2, but others made pledges elsewhere.

      Reply

3. Another thing reviewers can do is to encourage journals to adopt (as currently 9 already have https://osf.io/tvyxz/wiki/5.%20Adoptions%20and%20Endorsements/) the Open Data badge, the Open Methods badge, and the Registered badge (https://osf.io/tvyxz/wiki/home/) to indicate which articles have posted their data, methods, and preregistered their study.

   I am attempting to collect progressive and best reviewer practices here, please add:
   https://docs.google.com/document/d/1gG06IrUKJI2Nzul53txZ3W2HjzpzNlsJWOHFyuJ-HKc/edit

   Reply

   1. Hi Alex, by “please add” did you mean to add PRO to the doc, or to add a link to the doc to one of the PRO pages?

      Reply

      1. Sorry for the ambiguity, it was just a request for people to add their best reviewer practices to https://docs.google.com/document/d/1gG06IrUKJI2Nzul53txZ3W2HjzpzNlsJWOHFyuJ-HKc/edit
         I had already added the Openness initiative to it, and I wouldn’t presume to ask you to link to it from your initiative, and I don’t know if you have an appropriate place for that.

         Reply

4. Great initiative this!
   Does part 4 of the guideline also include code for running data analyses? There is much to gain here I believe when it comes to reproducibility, especially of methodologically oriented papers; as a reviewer I would also like to be able to reproduce certain results without having to code the statistics first.

   Reply

   1. Point 4 is really about requiring documentation for the things one shares. It doesn’t require code, just documentation *if* one releases code. We actually left out code in the Initiative for a very specific reason: many Universities actually the retain copyright of code written by employees. Hence, licensing of code is a gray area, because the researchers themselves may not technically own it. We didn’t want to encourage people to share things that they may not technically have the right to share. This does not mean you can’t ask for it; I think no University would enforce copyright on analysis code. That would be silly. But it is your right as a reviewer to ask for analysis or simulation code. We just left it out of the Initiative just to be extra safe.

      Reply

      1. A crucial point re supplying the code: if the submitted paper is presenting a computational model of anything, it cannot be properly reviewed unless the reviewer can try out the model. So the submission must include a URL via which a runnable version of the model is downloadable by the reviewer.

         One step towards this is at least to require that whenever such a paper is published the model is made publicly available. Psychological Review used to do this, but I don’t think it does so anymore. If this is not done, such work is literally unreplicable.

         Reply

         1. Agree mostly (aside from the pedantic point that it need not be literally unreplicable if model is deterministic and pseudo code is provided). We just didn’t include code in the Initiative due to the copyright issue. Reviewers should absolutely ask for code, in my opinion.

            A next step for us is to get Universities to clarify that their policies do not prevent releasing code. We’re thinking of ways to do this, including some boilerplate one can send to their University IP department asking for clarification. These policies need to change.

            Reply
5. Pingback: New movement aims to make open science the norm | The Horizons Tracker

6. This is a great initiative! I struggle for the transparency in science and especially in psychology for years. The problem of raw data was mined to daylight over half a century by Leoy Wolins, a psychologist from Iowa State University who authorized one of his students to write a letter to 37 authors of original research articles, asking them to submit the raw data that their studies were based on. 32 out of 37 authors replied to the request. However, 21 of those 32 researchers who replied informed “with tremendous regret” that their data was accidentally destroyed, lost or archived in such a way that it was impossible to retrieve them. Only nine researchers, (i.e. 24% of the initial group) appeared to be willing to make their data available. Doctor Wolins, (an expert in statistics), took a closer look at the data received and concluded that only seven of them met requirements of something that we might call a reliable statistical analysis. Wolins, surprised by the scientists’ responses, described the experience in the American Psychologist, thus triggering a heated scientific debate over the availability of raw data.
   Since then many replicated this “experiment.” Inspired by the examples of Wolins and other researchers, I also decided to replicate their studies on my own. In order to do so, I randomly selected 50 empirical studies, all of which had been published in the last 12 months. I requested that the authors share the raw data from their studies. Out of fifty requests, I received 27 replies, which accounted for 54% of the entire sample. I received only seven raw data sets. A further seven responses could be labeled as “willing to cooperate.” My success rate was 30% of the total sample. It is remarkably close to the results presented by Wolins in 1962, and subsequently by Wicherts et al. in 2006.
   I wrote a commentary article about my study and submitted it to American Psychologist, where discussion of this particular topic started over half a century ago. To my disbelief, I was told that this particular subject has actually been completely explored, and further discussions from now on will be closed: “As you know, every member of APA receives American Psychologist. The limited space we have available constrains us, and we can accept only articles that are of interest to a broad range of psychologists and that have broad consequences for the science and practice of psychology. … the American Psychologist has recently published a comment on this topic, and the present proposed comment does not really add any new additional information to the discussion.”
   In the light of such explanations, it is nothing unusual that Gert Storms has been forced to resign after his refusal to review papers without raw data. How does one dare to reopen an issue already resolved once and for all? Fortunately, Gert Storms is not alone!
   If you are interested in detailed description of the problem of availability of raw data see also my book: T. Witkowski & M. Zatonski “Psychology Gone Wrong: THe dark Sides of Science and Therapy”.

   Reply
7. Pingback: Why We're Encouraging Authors to Share Their Data with Reviewers - Social Science Space
8. Pingback: The Peer Reviewers’ Openness Initiative, Year 1 – The Mnemonic Lode

9. Not sure if signed up. Can’t find name in signatories but when I tried to sign up it said email already in use.

   Meanwhile have preliminary pre-print that includes discussion about barriers to Open Science. Indpendently made same recommendation for revieewers as here. Do NOT refuse to review. Suggest major revisions to include open data without reading unopen version

   Reply

10. See pre-print
    Kornbrot, D. E. (2020). Share It! Guide and Rationale for Deposit for Open Transparent Replicable Scientific Research. PsyArXiv(May, 2020). https://doi.org/10.31234/osf.io/bgk32

    Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.