Large-scale replication projects of the last years (e.g., ManyLabs I, II, and III, Reproducibility Project: Psychology) showed that the “replication crisis” in psychology is more than just hot air: According to recent estimates, ~60% of current psychological research is not replicableI will not go into details here about 'What counts as a replication?'. The 60% number certainly can be debated on many grounds, but the take-home-message is: It's devastating.. This spurred a lot of developments, such as the TOP guidelines, which define transparency and openness criteria for scientific publications.
The field is thinking about how we can ensure that we generate more actual knowledge and less false positives, or in the words of John Ioannidis: How to make more published research true.
In order to fathom potential consequences for our own department of psychology at the Ludwig-Maximilians-UniversitĂ€t MĂŒnchen, our department’s administration unanimously decided to establish an Open Science Committee (OSC).
The committee’s mission and goals include:
- Monitor the international developments in the area of open science and communicate them to the department.
- Organize workshops that teach skills for open science (e.g., How do I write a good pre-registration? What practical steps are necessary for Open Data? How can I apply for the Open Science badges?, How to do an advanced power analysis, What are Registered Reports?).
- Develop concrete suggestions concerning tenure-track criteria, hiring criteria, PhD supervision and grading, teaching, curricula, etc.
- Channel the discussion concerning standards of research quality and transparency in the department. Even if we share the same scientific values, the implementations might differ between research areas. A medium-term goal of the committee is to explore in what way a department-wide consensus can be established concerning certain points of open science.
The OSC developed some first suggestions about appropriate actions that could be taken in response to the replication crisis at the level of our department. We focused on five topics:
- Supervision and grading of dissertations
- Voluntary public commitments to research transparency and quality standards (this also includes supervision of PhDs and coauthorships)
- Criteria for hiring decisions
- Criteria for tenure track decisions
- How to allocate the department’s money without setting incentives for p-hacking
Raising the bars naturally provokes backlashs. Therefore we emphasize three points right from the beginning:
- The described proposals are no âfinal programâ, but a basis for discussion. We hope these suggestions will trigger a discussion within research units and the department as a whole. Since the proposal targets a variety of issues, of course they need to be discussed in the appropriate committees before any actions are taken.
- Different areas of research differ in many aspects, and the actions taken can differ betweens these areas. Despite the probably different modes of implementation, there can be a consensus regarding the overarching goal – for example, that studies with higher statistical power offer higher gains in knowledge (ceteris paribus), and that research with larger gains in knowledge should be supported.
- There can be justified exceptions from every guideline. For example, some data cannot sufficiently be anonymized, in which case Open Data is not an option. The suggestions described here should not be interpreted as chains to the freedom of research, but rather as a statement about which values we as a research community represent and actively strive for.
Two chairs are currently developing a voluntary commitment to research transparency and quality standards. These might serve as a blue-print or at least as food for thought for other research units. When finished, these commitments will be made public on the department’s website (and also on this blog). Furthermore, we will collect our suggestions, voluntary commitments, milestones,Â etc. on a public OSF project.
Do you have an Open Science Committee or a similar initiative at your university? We would love to bundle our efforts with other initiatives, share experiences, material, etc. Contact us!
— Felix SchĂ¶nbrodt, Moritz Heene, Michael Zehetleitner, Markus Maier
Stay tuned – soon we will present a first major success of our committee!
(Follow me on Twitter for more updates on #openscience and our Open Science Committee: @nicebread303)
[Update 2015/1/14: I consolidate feedback from Twitter, comments, email, and real life into the main text (StackExchange-style), so that we get a good and improving answer. Thanks to @TonyLFreitas,Â @PhDefunct, @bahniks, @JoeHilgard, @_r_c_a, @richardmorey, @R__INDEX, the commenters at the end of this post and on the OSF mailing list, and many others for their feedback!]
In a recent lecture I talked about the replication crisis in psychology. After the lecture my students asked: âWe learn so much stuff in our lectures, and now you tell us that a considerable proportion of these âfactsâ probably are just false positives, or highly exaggerated? Then, what can we believe at all?â. A short discussion soon led to the crucial question:
In the era of #repligate: What are validÂ cues for the trustworthinessÂ of a study?
Valid cues for trustworthiness of a single study:
- Pre-registration. This might be one of the strongest cues for trustworthiness. Pre-registration makes p-hacking and HARKing unlikely (Wagenmakers, Wetzels, Borsboom, Maas, & Kievit, 2012), and takes care for a sufficient amount of statistical power (At least, some sort of sample size planning has been done. Of course, this depends on the correctness of the a-priori effect size estimate).
- Sample size / Statistical Power. Larger samples mean higher power, higher precision, and less false positives (Bakker, van Dijk, & Wicherts, 2012; Maxwell, Kelley, & Rausch, 2008; SchĂ¶nbrodt & Perugini, 2013). Of course sample size alone is not a panacea. As always, the garbage in/garbage out principle holds, and a well designed lab study with n=40 can be much more trustworthy than a sloppy mTurk study with n=800. But allÂ other things being equal, I put more trust in larger studies.
- Independent high-power replications. If a study has been independently replicated from another lab with high power and preferably pre-registered, this probably is the strongest evidence for the trustworthiness of a study (How to conduct a replication? See the Replication Recipe by Brandt et al., 2014).
- I guess that studies with Open Data and Open Material have a higher replication rate
- “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Resultsâ (Wicherts, Bakker, & Molenaar, 2011) â> this is not exactly Open Data, because here authors only shared data upon request (orÂ not). But it points into the same direction.
- Beyond publishing Open Data at all, the neatness of the data set and the quality of the analysis script is an indicator (see also comment by Richard Morey). The journal “Quarterly Journal of Political Science” demands to publish raw data and analysis code that generates all the results reported in the paper. Of these submissions, 54% “had results in the paper that differed from those generated by the authorâs own code”! My fear is that analytical code that has not been refined and polished for publishing contains even more errors (not to speak of unreproducible point-and-click analyses). Therefore, a well prepared data set and analysis code should be a valid indicator.
- Open Material could be an indicator that people are not afraid of replications and further scrutiny
- An abstract with reasonable conclusions that stick close to the data – see also below: “Red flags”. This includes visible efforts of the authors to explain how they could be wrong and what precautions were/were not taken.
- A sensitivity analysis, which shows that conclusions do not depend on specific analytical choices. For Bayesian analyses this means to explore how the conclusions depend on the choice of the prior. But you could also show how your results change when you do not exlude the outliers, or do not apply that debatable transformation to your data (see also comment)
- Using the “21 Word Solution” of Simmons, Nelson, & Simonsohn (2012) leads to a better replication index.
Valid cues for trustworthiness of a research programme/ multiple studies:
Valid cues for UNtrustworthiness of a single study/ red flags:
In a comment below, Dr. R introduced the idea of “red flags”, which I really like. These red flags aren’t a prove of the untrustworthiness of a study – but definitely a warning sign to look closer and to be more sceptical.
- Sweeping claims, counterintuitive, and shocking results (that don’t connect to the actual data)
- Most p values are in the range of .03 – .05 (or, equivalently, most t-values in the 2-3 range, or most F-values are in the 4-9 range; see comment by Dr. R below).
- How does a distribution of p values look like when there’s an effect? See DaniĂ«l Lakens blog. With large samples, p-values just below .05 even indicate support for the null!
- It’s a highly cited result, but no direct replications have been published so far. That could be an indicator that many unsuccessful replication attempts went into the file-drawer (see comment by Ruben below).
- Too good to be true: If several low-power studies are combined in a paper, it can be very unlikely that all of them produce significant results. The “Test of Excess Significance” has been used to formally test for “too many significant results”. Although this formal test has been criticized (e.g., see The Etz-Files, and especially the long thread of comments, or this special issue on the test), I still think excess significance can be used as a red flag indicator to look closer.
Possibly invalid cuesÂ (cues which are often used, but only seemingly are indicators for a studyâs trustworthiness):
- The journalâs impact factor. Impact factors correlate with retractions (Fang & Casadevall, 2011), but do not correlate with a single paper’s citation count (see here).
- Iâm not really sure whether that is a valid or invalid cue for a study’s quality. The higher retraction rate might due to the stronger public interest and a tougher post-publication review of papers in high-impact journals. The IF seems not to be predictive of a single paper’s citation count; but I’m not sure either whether the citation count is an index of a study’s quality. Furthermore, “Impact factors should have no place in grant-giving, tenure or appointment committees.” (ibid.), see also a reccent article by @deevybee in Times Higher Education.
- On the other hand, the current replicability estimate of a full volume of JPSP is only at 20-30% (see Reproducibility Project: Psychology). A weak performance for one of our “best journals”.
- The authorâs publication record in high-impact journals or h-index. This might be a less valid cue as expected, or even an invalid cue.
- Meta-analyses. Garbage-in, garbage-out: Meta-analyses of a biased literature produce biased results. Typical correction methods do not work well. When looking at meta-analyses, at least one has to check whether and how it was corrected for publication bias.
“If a study has a large sample size, Open Data, and maybe even has been pre-registered, I would put quite some trust into the results.Â If the study has been independently replicated, even better. In contrast to common practice, I do not care so much whether this paper has been published in a high-impact journal or whether the author has a long publication record.Â The next step, of course, is: Read the paper, and judge itâs validity and the quality of its arguments!”
This list certainly is not complete, and I would be interested in your ideas, additions, and links to relevant literature!