A voluntary commitment to research transparency

The Reproducibility Project: Psychology was published last week, and it was another blow to the overall credibility of the current research system’s output.
Some interpretations of the results were in a “Hey, it’s all fine; nothing to see here; let’s just do business as usual” style. Without going into details about the “universal hidden moderator hypothesis” (see Sanjay’s blog for a reply) or “The results can easily explained by regression to the mean” (see Moritz’ and Uli’s reply): I do not share these optimistic views, and I do not want to do “business as usual”.
What makes me much more optimistic about the state of our profession than unfalsifiable post-hoc “explanations” is that there has been considerable progress towards an open science, such as the TOP guidelines for transparency and openness in scientific journals, the introduction of registered reports, or the introduction of the open science badges (Psych Science has increased sharing of data and materials from near zero to near 25%38% in 1.5 years, simply by awarding the badges). And all of this happend within the last 3 years!
Beyond these already beneficial changes, we asked ourself: What can we do on the personal and local department level to make more published research true?
A first reaction was the foundation of our local Open Science Committee (more about this soon). As another step, I developed together with some colleagues a Voluntary Commitment to Research Transparency.
The idea of that public commitment is to signal to others that we follow these guidelines of open science. The signal is supposed to go to:

  • Colleagues in the department and other universities (With the hope that more and more will join)
  • Co-authors (This is how we will do science)
  • Funding agencies (We prefer quality over quantity)
  • Potential future employers (This is our research style, if you want that)
  • PhD students:
    • If you want to do your PhD here: these are the conditions
    • If you apply for a job after your PhD, you will get the open-science-reputation-badge from us.

 
Now, here’s the current version of our commitment:
[Update 2015/11/19: I uploaded a minor revision which reflects some feedback from new signatories]

Voluntary Commitment to Research Transparency and Open Science

We embrace the values of openness and transparency in science. We believe that such research practices increase the informational value and impact of our research, as the data can be reanalyzed and synthesized in future studies. Furthermore, they increase the credibility of the results, as independent verification of the findings is possible.

Here, we express a voluntary commitment about how we will conduct our research. Please note that to every guideline there can be justified exceptions. But whenever we deviate from one of the guidelines, we give an explicit justification for why we do so (e.g., in the manuscript, or in the README file of the project repository).

As signatories, we warrant to follow these guidelines from the day of signature on:

Own Research

  1. Open Data: Whenever possible, we publish, for every first-authored empirical publication, all raw data which are necessary to reproduce the reported results on a reliable repository with high data persistence standards (such as the Open Science Framework).

  2. Reproducible scripts: For every first authored empirical publication we publish reproducible data analysis scripts, and, where applicable, reproducible code for simulations or computational modeling.

  3. We provide (and follow) the “21-word solution” in every empirical publication: “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”1 If necessary, this statement is adjusted to ensure that it is accurate.

  4. As co-authors we try to convince the respective first authors to act accordingly.

Reviewers

  1. As reviewers, we add the “standard reviewer disclosure request”, if necessary (https://osf.io/hadz3/). It asks the authors to add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes.

  2. As reviewers, we ask for Open Data (or a justification why it is not possible).2

Supervision of Dissertations

  1. As PhD supervisors we put particular emphasis on the propagation of methods that enhance the informational value and the replicability of studies. From the very beginning of a supervisor-PhD student relationship we discuss these requirements explicitly.

  2. From PhD students, we expect that they provide Open Data, Open Materials and reproducible scripts to the supervisor (they do not have to be public yet).

  3. If PhD projects result in publications, we expect that they follow points I. to III.

  4. In the case of a series of experiments with a confirmatory orientation, it is expected that at least one pre-registered study is conducted with a justifiable a priori power analysis (in the frequentist case), or a strong evidence threshold (e.g., if a sequential Bayes factor design is implemented). A pre-registration consists of the hypotheses, design, data collection stopping rule, and planned analyses.

  5. The grading of the final PhD thesis is independent of the studies’ statistical significance. Publications are aspired; however, a successful publication is not a criterion for passing or grading.

Service to the field

  1. As members of committees (e.g., tenure track, appointment committees, teaching, professional societies) or editorial boards, we will promote the values of open science.

1Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012). A 21 word solution. Retrieved from: http://dx.doi.org/10.2139/ssrn.2160588

2See also Peer Reviewers’ Openness Initiative: http://opennessinitiative.org/

So far, 4 members of our department, and 8 researchers from other universities have signed the commitment – take us at our word!
We hope that many more will join the initiative, or think about crafting their own personal commitment, at the openness level they feel comfortable with.

9 thoughts on “A voluntary commitment to research transparency”

  1. This seems like a no-brainer. How does one from another university than LMU sign up?
    But one thing I don’t understand is: why do we have to be restricted to putting up our data on the OSF? I prefer to put up my data+code next to the pdf on my home page. This gives me full control over the version I have, and full control over where it is stored. I tried storing my data etc. on a more general-purpose portal, but it was extremely tedious to upload each paper’s materials separately, and the portal was buggy so that I ended up with all kinds of technical problems like duplicate copies of my paper etc, which could never be removed.
    Currently I have a single bash script that auto-uploads my papers, code+data etc. to my home page, updates my home page as well as my cv as pdf, and that’s it. As soon as I have published a paper and have a zip archive with the data+code, uploading it to my home page is trivial, I just run the script. But all these portals require me to log-in (which is usually preceded by my forgetting the password, wasting even more time), input all kinds of information, and then rely on them to host my materials. I don’t see the point of that.
    Is it possible to broaden the statement to allow the data to be hosted *some*where publicly? Lately I have started hosting data and code (and lecture notes—pretty much everything) on github; this doesn’t affect my workflow as I just change the bash script to reflect the location of the data.

    1. PS: Your automated reproducibility setup with batch scripts is admirable!

  2. Hi Shravan,
    > How does one from another university than LMU sign up?
    Actually, we could simply remove the LMU logo. I will do that, and we are very happy if more researchers from any institution join as signatories! Well, signing up simply works by sending an email to me 😉
    > why do we have to be restricted to putting up our data on the OSF?
    Oh, that is just our personal preference. We put into the first footnote: “Or a comparable reliable repository with high data persistence standards.”. If you consider your own website as such a repository, that’s fine! We just experienced too often that data files etc. vanish from personal homepages and therefore prefer a storage with long-term availability and persistent links.
    > Is it possible to broaden the statement to allow the data to be hosted *some*where publicly?
    I would think: yes. If our document gains some impact beyond our initial group, we can collect your feedback, discuss and work on a revised version.
    But we started this initiative deliberately as a local group; it was important to us to get this project going and not to discuss the details with a too large group. We refrained from creating a standardized, one-for-all commitment, as research fields differ from each other, and researcher differ in the level of openness they are comfortable with. So, I think we have to find a balance between a broad impact (We are happy about every signatory!) and the necessary personalization (We do not want to settle on a minimum compromise, but rather express our personal view of scientific values).
    Thanks for your feedback!

  3. Thank you very much for this initiative!
    I hope many other researchers will follow in your footsteps.

  4. Pingback: Anonymous

Comments are closed.