Several recent discussions on the Psychological Methods Facebook group surrounded the question whether an optional stopping procedure leads to biased effect size estimates (see also this recent blog post by Jeff Rouder). Optional stopping is a rather new technique, and potential users wonder about the potential down-sides, as these (out-of-context) statements demonstrate:

- “… sequential testing appears to inflate the observed effect size”
- “discussion suggests to me that estimation is not straight forward?“
- “researchers who are interested in estimating population effect sizes should not use […] optional stopping“
- “we found that truncated RCTs provide biased estimates of effects on the outcome that precipitated early stopping” (Bassler et al., 2010)

Hence, the concern is that the usefulness of optional stopping is severely limited, because of this (alleged) bias in parameter estimation.

Here’s a (slightly shortened) scenario from a Facebook discussion:

Given the recent discussion on optional stopping and Bayes, I wanted to solicit opinions on the following thought experiment.Researcher A collects tap water samples in a city, tests them for lead, and stops collecting data once a t-test comparing the mean lead level to a “safe” level is significant atp<.05. After this optional stopping, researcher A computes a Bayesian posterior (with weakly informative prior), and reports the median of the posterior as the best estimate of the lead level in the city.Researcher B collects the same amount of water samples but with a pre-specified N, and then also computes a Bayesian estimate.Researcher C collects water samples from every single household in the city (effectively collecting the whole population).Hopefully we can all agree that the best estimate of the mean lead level in the city is obtained by researcher C. But do you think that the estimate of researcher B is closer to the one from researcher C and should be preferred over the estimate of researcher A? What – if anything – does this tell us about optional stopping and its influence on Bayesian estimates?

Let’s simulate the scenario (R code provided below) with the following settings:

- The true lead level in the city has a mean of 3 with a SD of 2
- The “safe” lead level is defined at 2.7 (or below)

We run 10,000 studies with strategy A and save the sample mean of the lead level along with the final sample size. We run 10,000 studies with strategy B (sample sizes matched to those of the 10,000 A-runs) and save the sample mean of the lead level along with the sample size.

In contrast to the quoted scenario above, I will *not* compute a Bayesian posterior, because the usage of a prior *will* bias the estimate. (A side note: When using a prior, this bias is deliberately accepted, because a small bias is traded in for a reduction in variance of the estimates, as extreme and implausible sample estimates are shrunken towards more realistic numbers). Here, we simply take the plain sample mean, because this is an unbiased estimator – at least in the typical textbook-case of fixed sample sizes. But what happens with optional stopping?

For strategy A, we compute the mean across all significant Monte Carlo simulations (which were ~11%), which is 1.42.

This is much less than the true value of 3! When we look at the 10,000 fixed-*n* studies with the same sample sizes, we get a mean lead level of 3.00, which is exactly the true value.

The impact of optional stopping seems devastating – it screws up my effect size estimates, and leads to an underestimation of the true lead level!

Does it really?

The naive analysis, however, ignores two crucial points:

**If effect sizes from samples with different sample sizes are combined, they must be meta-analytically weighted according to their sample size (or precision).**Optional stopping (e.g., based on*p*-values, but also based on Bayes factors) leads to a conditional bias: If the study stops very early, the effect size must be overestimated (otherwise it wouldn’t have stopped with a significant*p*-value). But early stops have a small sample size, and in a meta-analysis, these extreme early stops will get a small weight.**The determination of sample size (fixed vs. optional stopping) and the presence of publication bias are separate issues.**By comparing strategy A and B, two issues are (at least implicitly) conflated: A does optional stopping*and*has publication bias, as she only reports the result if the study hits the threshold. Non-significant results go into the file drawer. B, in contrast, has a fixed sample size, and reports all results,*without*publication bias. You can do optional stopping without publication bias (stop if significant, but also report result if you didn’t hit the threshold before reaching n_max). Likewise, if B samples a fixed sample size, but only reports trials in which the effect size is close to a foregone conclusion, it will be very biased as well.

As it turns out, the overestimations from early terminations are perfectly balanced by *under*estimations from late terminations (Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2015). Hence, optional stopping leads to a *conditional* bias (i.e., conditional on early or late termination), but it is *unconditionally* unbiased.

Hence, let’s keep these factors separate in our analysis and look at the 2 (publication bias: yes vs. no) x 2 (optional stopping vs. fixed sample size) x 2 (naive average vs. meta-analytically weighted average) combinations.

For this purpose, we have to update the simulation:

- The strategies A and B
*without publication bias*report all outcomes - Strategy A
*with*publication bias reports only the studies, which are significantly lower than the safe lead level - Strategy B
*with*publication bias reports only studies which show a sample mean which is smaller than the safe lead level (regardless of significance)

This is the sampling distribution of the sample means, across all 10,000 replications:

The distribution from strategy B (fixed-*n*) is well-behaved and symmetric. The distribution from strategy A (optional stopping) shows a bump at small effect sizes (these are the early stops with a small lead level).

Another way to look at this is to plot the single study estimates by sample size:

Early terminations in the sequential design underestimate the true level – but the late terminations at n=50 overestimate on average in the sequential design. This is the conditional bias – underestimation in early stops (because the optional stopping favored small lead levels), but overestimation in late stops. In the fixed-*n* design there is no conditional bias.

Here are the compute mean levels in our 8 combinations (true value = 3):

Sampling plan | PubBias | Naive mean | Weighted mean |
---|---|---|---|

sequential | FALSE | 2.85 | 3.00 |

fixed | FALSE | 3.00 | 3.00 |

sequential | TRUE | 1.42 | 1.71 |

fixed | TRUE | 2.46 | 2.55 |

If you selectively only report studies with a desired outcome (rows 3 & 4), the estimates cannot be trusted – all of them are way below the true value. Or, as Joachim Vandekerckhove put it: “I think it’s obvious that you can’t actively bias your data and expect magic to happen”.

If you report all studies (no publication bias), they must be properly weighted if they are combined. And then it does not matter whether sample sizes are fixed or optionally stopped! Both sampling plans lead to unbiased estimates.

(To be precise: it does not matter with respect to the unbiasedness of effect size estimates. It does matter concerning other properties, like the variance of estimates or the average sample size).

A more detailed analysis of the impact of sequential testing on parameter estimates can be found in our paper “Sequential Hypothesis Testing with Bayes Factors“. Finally, I want to quote a paragraph from our recent paper on Bayes Factor Design Analysis (Schönbrodt & Wagenmakers, 2016), which also summarizes the discussion and provides some more references:

Concerning the sequential procedures described here, some authors have raised concerns that these procedures result in biased effect size estimates (e.g., Bassler et al., 2010, J. Kruschke, 2014). We believe these concerns are overstated, for at least two reasons.First, it is true that studies that terminate early at the H1 boundary will, on average, overestimate the true effect. This conditional bias, however, is balanced by late terminations, which will, on average, underestimate the true effect. Early terminations have a smaller sample size than late terminations, and consequently receive less weight in a meta-analysis. When all studies (i.e., early and late terminations) are considered together, the bias is negligible (Berry, Bradley, & Connor, 2010; Fan, DeMets, & Lan, 2004; Goodman, 2007; Schönbrodt et al., 2015). Hence, the sequential procedure is approximately unbiased overall.Second, the conditional bias of early terminations is conceptually equivalent to the bias that results when only significant studies are reported and non-significant studies disappear into the file drawer (Goodman, 2007). In all experimental designs –whether sequential, non-sequential, frequentist, or Bayesian– the average effect size inevitably increases when one selectively averages studies that show a larger-than-average effect size. Selective publishing is a concern across the board, and an unbiased research synthesis requires that one considers significant and non-significant results, as well as early and late terminations.Although sequential designs have negligible unconditional bias, it may nevertheless be desirable to provide a principled “correction” for the conditional bias at early terminations, in particular when the effect size of a single study is evaluated. For this purpose, Goodman (2007) outlines a Bayesian approach that uses prior expectations about plausible effect sizes. This approach shrinks extreme estimates from early terminations towards more plausible regions. Smaller sample sizes are naturally more sensitive to prior-induced shrinkage, and hence the proposed correction fits the fact that most extreme deviations from the true value are found in very early terminations that have a small sample size (Schönbrodt et al., 2015).

[cc lang=”rsplus” escaped=”true” collapse=”true” ]

library(ggplot2)

library(dplyr)

library(htmlTable)

]]>library(ggplot2)

library(dplyr)

library(htmlTable)

# set seed for reproducibility

set.seed(0xBEEF)

trueLevel <- 3 trueSD <- 2 safeLevel <- 2.7 maxN <- 50 minN <- 3 B <- 10000 # number of Monte Carlo simulations res <- data.frame() for (i in 1:B) { print(paste0(i, "/", B)) maxSample <- rnorm(maxN, trueLevel, trueSD) # optional stopping for (n in minN:maxN) { t0 <- t.test(maxSample[1:n], mu=safeLevel, alternative="less") #print(paste0("n=", n, "; ", t0$estimate, ": ", t0$p.value)) if (t0$p.value <= .05) break; } finalSample.seq <- maxSample[1:n] # now construct a matched fixed-n finalSample.fixed <- rnorm(n, trueLevel, trueSD) # --------------------------------------------------------------------- # save results in long format # sequential design res <- rbind(res, data.frame( id = i, type = "sequential", n = n, p.value = t0$p.value, selected = t0$p.value <= .05, empMean = mean(finalSample.seq) )) # fixed design res <- rbind(res, data.frame( id = i, type = "fixed", n = n, p.value = NA, selected = mean(finalSample.fixed) <= safeLevel, # some arbitrary publication bias selection empMean = mean(finalSample.fixed) )) } save(res, file="res.RData") # load("res.RData") # Figure 1: Sampling distribution ggplot(res, aes(x=n, y=empMean)) + geom_jitter(height=0, alpha=0.15) + xlab("Sample size") + ylab("Sample mean") + geom_hline(yintercept=trueLevel, color="red") + facet_wrap(~type) + theme_bw() # Figure 2: Individual study estimates ggplot(res, aes(x=empMean)) + geom_density() + xlab("Sample mean") + geom_vline(xintercept=trueLevel, color="red") + facet_wrap(~type) + theme_bw() # the mean estimate of all late terminations res %>% group_by(type) %>% filter(n==50) %>% summarise(lateEst = mean(empMean))

# how many strategy A studies were significant?

res %>% filter(type==”sequential”) %>% .[[“selected”]] %>% table()

# Compute estimated lead levels

est.noBias <- res %>% group_by(type) %>% dplyr::summarise(

bias = FALSE,

naive.mean = mean(empMean),

weighted.mean = weighted.mean(empMean, w=n)

)

est.Bias <- res %>% filter(selected==TRUE) %>% group_by(type) %>% dplyr::summarise(

bias = TRUE,

naive.mean = mean(empMean),

weighted.mean = weighted.mean(empMean, w=n)

)

est <- rbind(est.noBias, est.Bias) est # output a html table est.display <- txtRound(data.frame(est), 2, excl.cols=1:2) t1 <- htmlTable(est.display, header = c("Sampling plan", "PubBias", "Naive mean", "Weighted mean"), rnames = FALSE) t1 cat(t1) [/cc]

Related posts:

]]>University hiring decisions often are driven (amongst other criteria) by publication quantity and journal prestige: “Several universities base promotion decisions on threshold *h*-index values and on the number of articles in ‘high-impact’ journals” (Hicks, Wouters, Waltman, de Rijcke, & Rafols, 2015), and Nosek, Spies, & Motyl (2012) mention “[…] the prevailing perception that publication numbers and journal prestige are the key drivers for professional success”.

We all know where this focus on pure quantity and too-perfect results led us: “In a world where researchers are rewarded for how many papers they publish, this can lead to a decrease in the truth value of our shared knowledge” (Nelson, Simmons, & Simonsohn, 2012), which can be seen in ongoing debates about low replication rates in psychology, medicine, or economics.

Doing studies with high statistical power, preparing open data, and trying to publish realistic results that are not hacked to (unrealistic) perfection will slow down scientists. Researchers engaging in these good research practices probably will have a smaller quantity of publications, and if that is the major selection criterion, they have a disadvantage in a competitive job market for tenured positions.

**For this reason, hiring standards have to change as well towards a valuation of research transparency, and the department of psychology at LMU München did the first step into this direction.**

Based on a suggestion of our Open Science Committee, the department added a paragraph to a professorship job advertisement which asks for an open science statement from the candidates:

Here’s a translation of the open science paragraph:

Our department embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open material, and pre-registrations. Candidates are asked to describe in what way they already pursued and plan to pursue these goals.

This paragraph clearly communicates open science as a core value of our department.

Of course, criteria of research transparency will not be the only criteria of evaluation for candidates. But, to my knowledge, this is the first time that they are *explicit* criteria.

Jean-Claude Burgelman (Directorate General for Research and Innovation of the European Commission) says that “the career system has to gratify open science”. I hope that many more universities will follow the LMU’s lead with an explicit commitment to open science in their hiring practices.

]]>Beyond doubt, change has to occur at the institutional level. In particular, many journals have already done a lot (see, for example, the TOP guidelines or the new registered reports article format). But journal policies aren’t enough, particularly since they are often not enforced.

In this blog post, I want to advocate for a complementary position of agency and empowerment: Let’s focus on steps *each individual* can do!

Here I want to show 9 steps that each individual can do, starting today, to foster open science:

1) **Join the community**. Follow open science advocates on Twitter and blogs. While monitoring these tweets does not change anything per se, it can give you important updates about developments in open science, and useful hints about how to implement open science in practice. Here’s my personal, selective, and incomplete list of Twitter users that frequently tweet about open science: https://twitter.com/nicebread303/lists/openscience

2) **Engage open values in peer review.** I started to realize that my work as a reviewer is very valuable work. I review more than 6x the number of papers that I submit myself. I receive more requests than I can handle, so I have to decide anyway which request to accept and which not. Where should I allocate my reviewing resources to? I prefer not to allocate them to research that is closed and practically unverifiable. I’d rather allocate them to research that is transparent, verifiable, sustainable, and re-usable.

Exactly this is the goal of the PRO initiative (Peer Reviewer Openness initiative), which uses the reviewer’s role to foster open science practices. The vision of the initiative is to switch from an opt-in model to an opt-out model: Openness is the new default; if authors don’t want it, they have to explicitly opt out. Signatories of the initiative only provide a comprehensive review of a manuscript if (a) open data and open material is provided, or (b) a public justification is given in the manuscript why this is not possible. Since the two weeks of the initiative’s existence, more than 160 reviewers signed it. I think this group already can have some impact, and I hope that more will sign.

[Read the paper — Sign the Initiative — More resources for open science]

Previous posts on the PRO initiative by Richard Morey, Candice Morey, Rolf Zwaan, and Daniel Lakens

3) **Commit yourself to open science.** In our “Voluntary Commitment to Research Transparency and Open Science” we explain which principles of research transparency we will follow from the day of signature on (see also my blog post). If you like it, sign it, and show the world that your research can be trusted and verified. Or use it as an inspiration to craft your own transparency commitment, on the openness level that you feel comfortable with.

4) **Find local li****ke-minded people. **Find colleagues in your department that embrace the values of open science as you do. Found a local open science initiative where you can exchange about challenges, help each other with practical problems (How did that pre-registration work?), and talk about ways open science can be implemented in your specific field. Use this “coalition of the willing” as the starting point for the next step …

5) **Found a local Open-Science-Committee.** Explore whether your local open science initiative could be installed as an official open science committee (OSC) at your department/ institution. See our OSF project for information about our open science committee at the department psychology at LMU Munich. Maybe you can reuse and adapt some of our material. Not all of our faculty members have the same opinion about this committee, some are enthusiastic, some are more skeptical. But still, the department’s board unanimously decided to establish this committee in order to keep the discussion going. Our OSC has 32 members from all chairs and we meet two times each semester. Our OSC has 4 goals:

- Monitor the international developments in the area of open science and communicate them to the department.
- Organize workshops that teach skills for open science
- Develop concrete suggestions concerning tenure-track criteria, hiring criteria, PhD supervision and grading, teaching, curricula, etc.
- Channel the discussion concerning standards of research quality and transparency in the department. Explore in what way a department-wide consensus can be established concerning certain points of open science.

6) **Pre-register your next study**. Pre-registration is a new skill we have to learn, so the first try does not have to be perfect. For example, I had to revise two of my registrations because I forgot important parts in the first version. In my experience, writing a few pre-registration documents gives you a better feeling for how long they take, what they should contain, what level of detail is appropriate, etc.

You can even win 1000$ if you participate in the pre-reg challenge!

7) **Teach open science practices to students.** You could plan your next Research Methods course as a pre-registered replication study. See also this OSF collection of syllabi, the “Good Science, Bad Science” course from EJ Wagenmakers, and the OSF Collaborative Replications and Education Project (CREP).

8) **Submit a registered report.** Think about submitting a **registered report** if there’s a journal in your field that supports this format. In this new article format an introduction, methods section, and analysis plan is submitted *before* data is collected. This proposal is sent to review, and in the positive case you get an in-principle-acceptance and proceed to actual data collection. This means, the paper is published independent of the results (unless you screw up your data collection or analysis).

9) **Promote the values of open science in committees.** As a member of a job committee, you can argue for open science criteria and evaluate candidates (amongst other criteria, of course) whether they engage in open practices. For example, Betsy Levy Paluck wrote in her blog: “In a hiring capacity, I will appreciate applicants who, though they do not have a ton of publications, can link their projects to an online analysis registration, or have posted data and replication code. Why? I will infer that they were slowing down to do very careful work, that they are doing their best to build a cumulative science.”

These are 9 small and medium steps, which each researcher could implement to some extent. If enough researchers join us, we can change the face of research.

]]>Well, no. As you will see in a minute, the “false discovery rate” (aka. false-positive rate), which indicates the probability that a significant *p*-value actually is a false-positive, usually is much higher than 5%.

Oakes (1986) asked the following question to students and senior scientists:

You have a p-value of .01. Is the following statement true, or false?You know, if you decide to reject the null hypothesis, the probability that you are making the wrong decision.

The answer is “false” (you will learn why it’s false below). But 86% of all professors and lecturers in the sample who were teaching statistics (!) answered this question erroneously with “true”. Gigerenzer, Kraus, and Vitouch replicated this result in 2000 in a German sample (here, the “statistics lecturer” category had 73% wrong). Hence, it is a wide-spread error to confuse the *p*-value with the false discovery rate.

To answer the question “What’s the probability that a significant *p*-value indicates a true effect?”, we have to look at the **positive predictive value (PPV)** of a significant *p*-value. The PPV indicates the proportion of significant *p*-values which indicate a real effect amongst all significant *p*-values. Put in other words: *Given that a p-value is significant: What is the probability (in a frequentist sense) that it stems from a real effect?*

(The false discovery rate simply is 1-PPV: the probability that a significant *p*-value stems from a population with null effect).

That is, we are interested in a conditional probability Prob(effect is real | *p*-value is significant).

Inspired by Colquhoun (2014) one can visualize this conditional probability in the form of a tree-diagram (see below). Let’s assume, we carry out 1000 experiments for 1000 different research questions. We now have to make a couple of prior assumptions (which you can make differently in the app we provide below). For now, we assume that 30% of all studies have a real effect and the statistical test used has a power of 35% with an α level set to 5%. That is of the 1000 experiments, 300 investigate a real effect, and 700 a null effect. Of the 300 true effects, 0.35*300 = 105 are detected, the remaining 195 effects are non-significant false-negatives. On the other branch of 700 null effects, 0.05*700 = 35 *p*-values are significant by chance (false positives) and 665 are non-significant (true negatives).

This path is visualized here (completely inspired by Colquhoun, 2014):

Now we can compute the false discovery rate (FDR): 35 of (35+105) = 140 significant *p*-values actually come from a null effect. That means, 35/140 = 25% of all significant *p*-values do not indicate a real effect! That is much more than the alleged 5% level (see also Lakens & Evers, 2014, and Ioannidis, 2005)

Together with Michael Zehetleitner I developed an interactive app that computes and visualizes these numbers. For the computations, you have to choose 4 parameters.

Let’s go through the settings!

Some of our investigated hypotheses are actually true, and some are false. As a first parameter, we have to estimate what proportion of our investigated hypotheses is actually true.

Now, what is a good setting for the a priori proportion of true hypotheses? It’s certainly not near 100% – in this case only trivial and obvious research questions would be investigated, which is obviously not the case. On the other hand, the rate can definitely drop close to zero. For example, in pharmaceutical drug development “only one in every 5,000 compounds that makes it through lead development to the stage of pre-clinical development becomes an approved drug” (Wikipedia). Here, only 0.02% of all investigated hypotheses are true.

Furthermore, the number depends on the field – some fields are highly speculative and risky (i.e., they have a low prior probability), some fields are more cumulative and work mostly on variations of established effects (i.e., in these fields a higher prior probability can be expected).

But given that many journals in psychology exert a selection pressure towards novel, surprising, and counter-intuitive results (which a priori have a low probability of being true), I guess that the proportion is typically lower than 50%. My personal grand average gut estimate is around 25%.

(Also see this comment and this reply for a discussion about this estimate).

That’s easy. The default α level usually is 5%, but you can play with the impact of stricter levels on the FDR!

The average power in psychology has been estimated at 35% (Bakker, van Dijk, & Wicherts, 2012). An median estimate for neuroscience is at only 21% (Button et al., 2013). Even worse, both estimates can be expected to be inflated, as they are based on the average *published* effect size, which almost certainly is overestimated due to the significance filter (Ioannidis, 2008). Hence, the average true power is most likely smaller. Let’s assume an estimate of 25%.

Finally, let’s add some realism to the computations. We know that researchers employ “researchers degrees of freedom”, aka. questionable research practices, to optimize their *p*-value, and to push a “nearly significant result” across the magic boundary. How many reported significant *p*-values would not have been significant without *p*-hacking? That is hard to tell, and probably also field dependent. Let’s assume that 15% of all studies are *p*-hacked, intentionally or unintentionally.

When these values are defined, the app computes the FDR and PPV and shows a visualization:

With these settings, **only 39% of all significant studies are actually true**!

Wait – what was the success rate of the Reproducibility Project: Psychology?** 36% of replication projects found a significant effect** in a direct replication attempt. Just a coincidence? Maybe. Maybe not.

The formula to compute the FDR and PPV are based on Ioannidis (2005: “Why most published research findings are false“). A related, but different approach, was proposed by David Colquhoun in his paper “An investigation of the false discovery rate and the misinterpretation of p-values” [open access]. He asks: “How should one interpret the observation of, say, *p*=0.047 in a *single* experiment?”. The Ioannidis approach implemented in the app, in contrast, asks: “What is the FDR in a *set of studies* with p <= .05 and a certain power, etc.?”. Both approaches make sense, but answer different questions.

- See also Daniel Laken’s blog post about the same topic, and the interesting discussion below it.
- Will Gervais also wrote a blog post on the topic and also has a cool interactive app which displays the results in another style.
- Commenter SamGG pointed to another publication which has illuminating graphics:
Krzywinski, M., & Altman, N. (2013). Points of significance: Power and sample size.
*Nature Methods*,*10*, 1139–1140. doi:10.1038/nmeth.2738

In the meeting, I first gave a quick overview about the replication crisis in psychology. I had the impression that we had a large consensus about the fact that we indeed have a problem, and that we should think about possible consequences. Then we started an open discussion where we collected questions, reservations, and ideas.

Here are some unordered topics of our discussions. Not all of them could be resolved at that meeting (which was not the goal), but eventually these stubs could result in a FAQ:

- It is important to acknowledge the
**diversity of our (sub)fields**. Even if we agree on the overarching values of open science, the specific implementations might differ. The current discussion often is focused on experimental laboratory (or online) research. What about existing large-scale data sets? What about sensitive video data from infant studies? What about I&O research in companies, where agreements with the work council forbid open data? Feasible protocols and solutions have to be developed in these fields. **Does the new focus on power lead to boring and low-risk “Mturk-research”**? This is certainly not the goal, but we should be aware that this could happen as an unintended side effect. For example, all ManyLab projects focused on easy-to-implement computer studies. Given the focus of the projects this is understandable; but we should not forget “the other” research.- We had a longer discussion which could be framed as “
**strategic choices vs. intrinsic motivation (and social mandate) to increase knowledge**”. From a moral point of view, the choice is clear. But we all are also individuals who have to feed our families (or, at least, ourselves), and the strategic perspective has an existential relevance to us.

Related to this question is also the next point: **What about the “middle” generation who soon will look for a job in academia?**Can we really recommend to go the open way? Without the possibility to*p*-hack, and with the goal to run high-powered studies (which typically have a larger*n*), individual productivity (aka: # of published papers) will decline. (Of course, “productivity” in terms of “increase in valid knowledge” will rise).

This would be my current answer:*I expect that the gain in reputation outweighs the potential loss in # of published papers.*Furthermore, we now have several techniques which allow us to assess the likelihood of*p*-hacking and the evidential value of a set of studies. If we present a paper with 4 studies and*p*s = .03, .04, .04, and .05, chances are high that we do not earn a lot of respect but rather sceptical frowns. Hence, with the increasing knowledge of healthy*p*-curves and other indicators, the old strategy of packing together too-good-to-be-true studies might soon backfire.

Finally, I’d advocate for an agentic position: It’s not some omnipotent external force that imposes an evil incentive structure on us.*We are the incentive structure!*At least at our department we can make sure that we do not incentivize massive*p*-hacking, but reward scientists that produce transparent, replicable, and honest research.- The new focus on replicability and transparency criteria does
*not*imply that other quality indicators (such as good theoretical foundations) are less important. - Some change can been achieved by positive, voluntary incentives. For example, the Open Science Badges led to 40% of papers having open data in the journal
*Psychological Science*. In other situations, we might need mandatory rules. Concerning this**voluntary/mandatory dimension**: When is which approach appropriate and more constructive? - An experience: Registered reports can take a long time in the review process – A problem for publication-based dissertations?
- We have to teach the field on what methods we base the conclusion that we have a replicability problem. In some discussions (not in our OSC ;-)) you can hear something like: “Some young greenhorns invent some fancy statistical index, and tell us that everything we have done is crap. First they should show that their method is valid!” It is our responsibility to explain the methods to researchers that do not follow the current replicability discussion so closely or are not so statistically savvy.
- Idea: Should we give an annual
**Open-Science-Award**at our department?

We have no ready-made answers for many of these questions. Most of them have to be tackled at multiple levels. Any comments and other perspectives on these open questions are appreciated!

One goal of the committee is to train our researchers in new tools and topics. I am happy to announce that we will host at least 4 talks/workshops in our department in the remainder of 2015:

- Sep 30, 2015: Jonathon Love (Amsterdam):
**JASP – A Fresh Way to Do Statistics (14-16, room 3322)** - Nov 5, 2015: Daniel Lakens (TU Eindhoven):
**Practical Recommendations to Increase the Informational Value of Studies** - End of November 2015 (TBA): Etienne LeBel:
**Introducing Curate Science (curatescience.org)** - Dec 2015 (TBA): Felix Schönbrodt:
**How to detect***p*-hacking and publication bias: A practical introduction to*p*-curve, R-index, etc.

The plan for the next meeting is to discuss our voluntary commitment to research transparency.

Some interpretations of the results were in a “Hey, it’s all fine; nothing to see here; let’s just do business as usual” style. Without going into details about the “universal hidden moderator hypothesis” (see Sanjay’s blog for a reply) or “The results can easily explained by regression to the mean” (see Moritz’ and Uli’s reply): I do not share these optimistic views, and I do not want to do “business as usual”.

What makes me much more optimistic about the state of our profession than unfalsifiable post-hoc “explanations” is that there has been considerable progress towards an open science, such as the TOP guidelines for transparency and openness in scientific journals, the introduction of registered reports, or the introduction of the open science badges (Psych Science has increased sharing of data and materials from near zero to near ~~25%~~38% in 1.5 years, simply by awarding the badges). And all of this happend within the last 3 years!

Beyond these already beneficial changes, we asked ourself: **What can we do on the personal and local department level to make more published research true?**

A first reaction was the foundation of our local Open Science Committee (more about this soon). As another step, I developed together with some colleagues a **Voluntary Commitment to Research Transparency**.

The idea of that public commitment is to signal to others that we follow these guidelines of open science. The signal is supposed to go to:

- Colleagues in the department and other universities (With the hope that more and more will join)
- Co-authors (This is how we will do science)
- Funding agencies (We prefer quality over quantity)
- Potential future employers (This is our research style, if you want that)
- PhD students:
- If you want to do your PhD here: these are the conditions
- If you apply for a job after your PhD, you will get the open-science-reputation-badge from us.

Now, here’s the current version of our commitment:

[Update 2015/11/19: I uploaded a minor revision which reflects some feedback from new signatories]

## Voluntary Commitment to Research Transparency and Open Science

We embrace the values of openness and transparency in science. We believe that such research practices increase the informational value and impact of our research, as the data can be reanalyzed and synthesized in future studies. Furthermore, they increase the credibility of the results, as independent verification of the findings is possible.

Here, we express a voluntary commitment about how we will conduct our research. Please note that to every guideline there can be justified exceptions. But whenever we deviate from one of the guidelines, we give an explicit justification for why we do so (e.g., in the manuscript, or in the README file of the project repository).

As signatories, we warrant to follow these guidelines from the day of signature on:## Own Research

Open Data: Whenever possible, we publish, for every first-authored empirical publication, all raw data which are necessary to reproduce the reported results on a reliable repository with high data persistence standards (such as the Open Science Framework).

Reproducible scripts: For every first authored empirical publication we publish reproducible data analysis scripts, and, where applicable, reproducible code for simulations or computational modeling.

We provide (and follow) the “21-word solution” in every empirical publication: “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”

^{1}If necessary, this statement is adjusted to ensure that it is accurate.As co-authors we try to convince the respective first authors to act accordingly.

## Reviewers

As reviewers, we add the “standard reviewer disclosure request”, if necessary (https://osf.io/hadz3/). It asks the authors to add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes.

As reviewers, we ask for Open Data (or a justification why it is not possible).

^{2}## Supervision of Dissertations

As PhD supervisors we put particular emphasis on the propagation of methods that enhance the informational value and the replicability of studies. From the very beginning of a supervisor-PhD student relationship we discuss these requirements explicitly.

From PhD students, we expect that they provide Open Data, Open Materials and reproducible scripts to the supervisor (they do not have to be public yet).

If PhD projects result in publications, we expect that they follow points I. to III.

In the case of a series of experiments with a confirmatory orientation, it is expected that at least one pre-registered study is conducted with a justifiable a priori power analysis (in the frequentist case), or a strong evidence threshold (e.g., if a sequential Bayes factor design is implemented). A pre-registration consists of the hypotheses, design, data collection stopping rule, and planned analyses.

The grading of the final PhD thesis is independent of the studies’ statistical significance. Publications are aspired; however, a successful publication is not a criterion for passing or grading.

## Service to the field

As members of committees (e.g., tenure track, appointment committees, teaching, professional societies) or editorial boards, we will promote the values of open science.

1Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012).

A 21 word solution. Retrieved from: http://dx.doi.org/10.2139/ssrn.21605882See also Peer Reviewers’ Openness Initiative: http://opennessinitiative.org/

So far, 4 members of our department, and 8 researchers from other universities have signed the commitment – **take us at our word!**

We hope that many more will join the initiative, or think about crafting their own personal commitment, at the openness level they feel comfortable with.

]]>Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize?Journal of Research in Personality,47, 609–612. doi:10.1016/j.jrp.2013.05.009

Interestingly (and in contrast to all of my other papers …), the paper has not only been cited in psychology, but also in medical chemistry, geophysical research, athmospheric physics, chronobiology, building research, and, most importantly, in the Indian Journal of Plant Breeding. Amazing.

And the best thing is: The paper is open access, and all simulation code and data are open on Open Science Framework. Use it and run your own simulations!

]]>The field is thinking about how we can ensure that we generate **more actual knowledge and less false positives**, or in the words of John Ioannidis: How to make more published research true.

In order to fathom potential consequences for our own department of psychology at the Ludwig-Maximilians-Universität München, our department’s administration unanimously decided to establish an **Open Science Committee (OSC)**.

The committee’s mission and goals include:

- Monitor the international developments in the area of open science and communicate them to the department.
- Organize workshops that teach skills for open science (e.g., How do I write a good pre-registration? What practical steps are necessary for Open Data? How can I apply for the Open Science badges?, How to do an advanced power analysis, What are Registered Reports?).
- Develop concrete suggestions concerning tenure-track criteria, hiring criteria, PhD supervision and grading, teaching, curricula, etc.
- Channel the discussion concerning standards of research quality and transparency in the department. Even if we share the same scientific values, the implementations might differ between research areas. A medium-term goal of the committee is to explore in what way a department-wide consensus can be established concerning certain points of open science.

The OSC developed some **first suggestions about appropriate actions** that could be taken in response to the replication crisis at the level of our department. We focused on five topics:

- Supervision and grading of dissertations
- Voluntary public commitments to research transparency and quality standards (this also includes supervision of PhDs and coauthorships)
- Criteria for hiring decisions
- Criteria for tenure track decisions
- How to allocate the department’s money without setting incentives for
*p*-hacking

Raising the bars naturally provokes backlashs. Therefore we emphasize three points right from the beginning:

*The described proposals are no “final program”, but a basis for discussion.*We hope these suggestions will trigger a discussion within research units and the department as a whole. Since the proposal targets a variety of issues, of course they need to be discussed in the appropriate committees before any actions are taken.*Different areas of research differ in many aspects, and the actions taken can differ betweens these areas.*Despite the probably different modes of implementation, there can be a consensus regarding the overarching goal – for example, that studies with higher statistical power offer higher gains in knowledge (ceteris paribus), and that research with larger gains in knowledge should be supported.*There can be justified exceptions from every guideline.*For example, some data cannot sufficiently be anonymized, in which case Open Data is not an option. The suggestions described here should not be interpreted as chains to the freedom of research, but rather as a statement about which values we as a research community represent and actively strive for.

Two chairs are currently developing a **voluntary commitment to research transparency and quality standards**. These might serve as a blue-print or at least as food for thought for other research units. When finished, these commitments will be made public on the department’s website (and also on this blog). Furthermore, we will collect our suggestions, voluntary commitments, milestones, etc. on a public OSF project.

Do you have an Open Science Committee or a similar initiative at your university? We would love to bundle our efforts with other initiatives, share experiences, material, etc. Contact us!

— Felix Schönbrodt, Moritz Heene, Michael Zehetleitner, Markus Maier

*Stay tuned – soon we will present a first major success of our committee!*

* (Follow me on Twitter for more updates on #openscience and our Open Science Committee: @nicebread303)*

Recently, I tried to approach the topic from an experiental perspective (“What does a Bayes factor feel like?“) by letting people draw balls from an urn and monitor the Bayes factor for an equal distribution of colors. Now I realized that I re-discovered an approach that Richard Royall did in his 1997 book “Statistical Evidence: A Likelihood Paradigm”: He also derived labels for likelihood ratios by looking at simple experiments, including ball draws.

But beyond this approach of getting an experiental access to LRs, all traditions mentioned above proposed in some way** labels or “grades” of evidence**.

These are summarized in my cheat sheet below.

(There’s also a PDF of the cheat sheet).

There’s considerable consensus about what counts as “strong evidence” (But this is not necessarily “independent replication” – maybe they just copied each other).

But there’s also the position that we **do not need labels at all** – the numbers simply speak for themselves! For an elaboration of that position, see Richard Morey’s blog post. Note that Kass & Raftery (1995) are often cited for their grades in the cheat sheet, but according to Richard Morey rather belong to the “need no labels” camp (see here and here). On the other hand, EJ Wagenmakers mentions that they use their guidelines themselves for interpretation and asks “when you propose labels and use them, how are you in the no-labels camp?”. Well, decide yourself (or ask Kass and Raftery personally), whether they belong into the “labels” or “no-labels” camp.

Now that I have some experience with LRs, I am inclined to follow the “no labels needed” position. But whenever I *explain* Bayes factors to people who are unacquainted with them, I really long for a descriptive label. I think the labels are short-cuts, which relieve you from the burden to explain how to interpret and judge an LR (You can decide yourself whether that is a good or a bad property of the labels).

To summarize, as LRs are not self-explanatory to the typical audience, I think you either need a label (which is self-explanatory, but probably too simplified and not sufficiently context-dependent), or you should give an introduction on how to interpret and judge these numbers correctly.

Burnham, K. P., & Anderson, D. R. (2002). *Model selection and multimodel inference: A practical information-theoretic approach*. Springer Science & Business Media.

Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. *Behavioral Ecology and Sociobiology*, *65*, 23–35. doi:10.1007/s00265-010-1029-6

Symonds, M. R. E., & Moussalli, A. (2011). A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike’s information criterion. *Behavioral Ecology and Sociobiology*, *65*, 13–21. doi:10.1007/s00265-010-1037-6

Good, I. J. (1985). Weight of evidence: A brief survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), *Bayesian Statistics 2* (pp. 249–270). Elsevier.

Jeffreys, H. (1961). *The theory of probability*. Oxford University Press.

Lee, M. D., & Wagenmakers, E.-J. (2013). *Bayesian cognitive modeling: A practical course*. Cambridge University Press.

Royall, R. M. (1997). *Statistical evidence: A likelihood paradigm*. London: Chapman & Hall.

Royall, R. M. (2000). On the probability of observing misleading statistical evidence. *Journal of the American Statistical Association*, *95*, 760–768. doi:10.2307/2669456

Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.

Morey, R. D. (2015). *On verbal categories for the interpretation of Bayes factors (Blog post).* http://bayesfactor.blogspot.de/2015/01/on-verbal-categories-for-interpretation.html

Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin & Review*, *16*, 225–237.