Archive for author

Gazing into the Abyss of P-Hacking: HARKing vs. Optional Stopping

by Angelika Stefan & Felix Schönbrodt

Almost all researchers have experienced the tingling feeling of suspense that arises right before they take a look at long-awaited data: Will they support their favored hypothesis? Will they yield interesting or even groundbreaking results? In a perfect world (especially one without publication bias), the cause of this suspense should be nothing else but scientific curiosity. However, the world, and specifically the incentive system in science, is not perfect. A lot of pressure rests on researchers to produce statistically significant results. For many researchers, statistical significance is the cornerstone of their academic career, so non-significant results in an important study can not only question their scientific convictions but also crash their hopes of professional promotion. (Although, fortunately things are changing for the better).

Now, what does a researcher do confronted with messy, non-significant results? According to several much-cited studies (for example John et al., 2012; Simmons et al., 2011), a common reaction is to start sampling again (and again, and again, …) in the hope that a somewhat larger sample size can boost significance. Another reaction is to wildly conduct hypothesis tests on the existing sample until at least one of them becomes significant (see for example: Simmons et al., 2011; Kerr, 1998 ). These practices, along with some others, are commonly known as p-hacking, because they are designed to drag the famous p-value right below the mark of .05 which usually indicates statistical significance. Undisputedly, p-hacking works (for a demonstration try out the p-hacker app). The two questions we want to answer in this blog post are: How does it work and why is that bad for science?

As many people may have heard, p-hacking works because it exploits a process called alpha error accumulation which is covered in most introductory statistics classes (but also easily forgotten again). Basically, alpha error accumulation means that as one conducts more and more hypothesis tests, the probability increases that one makes a wrong test decision at least once. Specifically, this wrong test decision is a false positive decision or alpha error, which means that you proclaim the existence of an effect although, in fact, there is none. Speaking in statistical terms, an alpha error occurs when the test yields a significant result although the null hypothesis (“There is no effect”) is true in the population. This means that p-hacking leads to the publication of an increased rate of false positive results, that is, studies that claim to have found an effect although, in fact, their result is just due to the randomness of the data. Such studies will never replicate.

At this point, the blog post could be over. P-Hacking exploits alpha error accumulation and fosters the publication of false positive results which is bad for science. However, we want to take a closer look at how bad it really is. In fact, some p-hacking techniques are worse than others (or, if you like the unscrupulous science villain perspective: some p-hacking techniques work better than others). As a showcase, we want to introduce two researchers: The HARKer takes existing data and conducts multiple independent hypothesis tests (based on multiple uncorrelated variables in the data set) with the goal to publish the ones that become significant. For example, the HARKer tests for each possible correlation in a large data set whether it differs significantly from zero. On the other hand, the Accumulator uses optional stopping. This means that he collects data for a single research question test in a sequential manner until either statistical significance or a maximum sample size is reached. For simplicity, we assume that they use neither other p-hacking techniques nor other questionable research practices.

The HARKer’s p-hacking strategy

Let us start with the HARKer: Since the conducted hypothesis tests in our defined scenario are essentially independent, the situation can be seen as a problem of multiple testing. This means, it is comparatively easy to determine the exact probability that the HARKer will end up with at least one false-positive result given a certain number of hypothesis tests. Assuming no effects in the population (for example, no correlation between the variables), one can picture the situation as a decision tree: At each branch level stands a hypothesis test which can either result in a non-significant result with 95% probability or in a (spurious) significant result with 5% probability, which is the \alpha level.

This figure shows a decision tree. It depicts alpha error accumulation with multiple testing / HARKing.

No matter how many hypothesis tests the HARKer conducts, there will only be one condition in the all-null scenario where no \alpha error occurs, that is, where all hypothesis tests yield non-significant results. The probability that this occurs can be calculated by 0.95 \cdot 0.95 \cdot 0.95 \cdot ... \cdot 0.95 = 0.95^x, with x being the number of conducted hypothesis tests. The probability that at least one of the hypothesis tests is significant is the probability of the complementary event, that is 1-(0.95^x). For example, when the HARKer computes 10 hypothesis with an alpha level of 0.05, the overall probability to obtain at least one false positive result is 1-(0.95^{10}) = 0.401. Of course, the formula can be adjusted for other suggested alpha levels, such as \alpha = 0.005 or \alpha = 0.1. We show this general formula in the R-code chunk below.

# The problem of HARKing

harker <- function(x, alpha){1-(1-alpha)^x}

The Accumulator’s p-hacking strategy

The Accumulator has a different tactic: Instead of conducting multiple hypothesis tests on different variables of one data set, he repeatedly conducts the same hypothesis test on the same variables in a growing sample. Starting with a minimum sample size, the Accumulator looks at the results of the analysis – if these are significant, data collection is stopped, if not, more data is collected until (finally) the results become significant, or a maximum sample size is reached. This strategy is also called Optional Stopping. Of course, the more often a researcher peeks at the data, the higher is the probability to obtain a false positive result at least once. However, this overall probability is not the same as the one obtained through HARKing. The reason is that the hypothesis tests are not independent in this case. Why is that? The same hypothesis test is repeatedly conducted on only slightly different data. In fact, the data that were used in in the first hypothesis test are used in every single of the subsequent hypothesis tests so that there is a spillover effect of the first test to every other hypothesis test in the set. Imagine, your initial sample contains an outlier: This outlier will affect the test results in any other test. With multiple testing, in contrast, the outlier will affect only the test in question but none of the other tests in the set.

So does this dependency make optional stopping more or less effective than HARKing? Of course, people have been wondering about this for quite a while. A paper by Armitage et al. (1969) demonstrates error accumulation in optional stopping for three different tests. We can replicate their results for the z-test with a small simulation (a more flexible simulation can be found at the end of the blog post): We start by drawing a large number of samples (iter) with the maximum sample size (n.max) from the null hypothesis. Then we conduct a sequential testing procedure on each of the samples, starting with a minimum sample size (n.min) and going up in several steps (step) up to the maximum sample size. The probability to obtain a significant result at least once up to a certain step can be estimated through the percentage of iterations that end up with at least one significant result at that point.

# The Problem of optional stopping

accumulator <- function(n.min, n.max, step, alpha=0.05, iter=10000){

# Determine places of peeks
peeks <- seq(n.min, n.max, by=step)

# Initialize result matrix (non-sequential)
res <- matrix(NA, ncol=length(peeks), nrow=iter)
colnames(res) <- peeks

# Conduct sequential testing (always until n.max, with peeks at pre-determined places)
for(i in 1:iter){
sample <- rnorm(n.max, 0, 1)
res[i,] <- sapply(peeks, FUN=function(x){sum(sample[1:x])/sqrt(x)})
}

# Create matrix: Which tests are significant?
signif <- abs(res) >= qnorm(1-alpha)

# Create matrix: Has there been at least one significant test in the trial?
seq.signif <- matrix(NA, ncol=length(peeks), nrow=iter)

for(i in 1:iter){
for(j in 1:ncol(signif)){
seq.signif[i,j] <- TRUE %in% signif[i, 1:j]
}
}

# Determine the sequential alpha error probability for the sequential tests
seq.alpha <- apply(seq.signif, MARGIN = 2, function(x){sum(x)/iter})

# Return a list of individual test p-values, sequential significance and sequential alpha error probability
return(list(seq.alpha = seq.alpha))

}

For example, the researcher conducts a two-sided one-sample z-test with an overall \alpha level of .05 in a sequential way. He starts with 10 observations, then always adds another 10 if the result is not significant, up to 100 observations at maximum. This means, he has 10 chances to peek at the data and end the data collection if the hypothesis test is significant. Using our simulation function, we can determine the probability to have obtained at least one false positive result at any of these steps:

set.seed(1234567)
res.optstopp <- accumulator(n.min=10, n.max=100, step=10, alpha=0.025, iter=10000)
print(res.optstopp[[1]])
[1] 0.0492 0.0824 0.1075 0.1278 0.1431 0.1574 0.1701 0.1788 0.1873 0.1949

We can see that with one single evaluation, the false positive rate is at the nominal 5%. However, when more in-between tests are calculated, the false positive rate rises to roughly 20% with ten peeks. This means that even if there is no effect at all in the population, the researcher would have stopped data collection with a signficant result in 20% of the cases.

A comparison of the HARKer’s and the Accumulator’s strategy

Let’s compare the false positive rates of HARKing and optional stopping: Since the researcher in our example above conducts one to ten dependent hypothesis tests, we can compare this to a situation where a HARKer conducts one to ten independent hypothesis tests. The figure below shows the results of both p-hacking strategies:

# HARKing False Positive Rates

HARKs <- harker(1:10, alpha=0.05)

p-hacking efficiency with optional stopping (as described in the blog post) and HARKing

We can see that HARKing produces higher false positive rates than optional stopping with the same number of tests. This can be explained through the dependency on the first sample in the case of optional stopping: Given that the null hypothesis is true, this sample is not very likely to show extreme effects in any direction (however, there is a small probability that it does). Every extension of this sample has to “overcome” this property not only by being extreme in itself but also by being extreme enough to shift the test on the overall sample from non-significance to significance. In contrast, every sample in the multiple testing case only needs to be extreme in itself. Note, however, that false positive rates in optional stopping are not only dependent on the number of interim peeks, but also on the size of the initial sample and on the step size (how many observations are added between two peeks?). The difference between multiple testing and optional stopping which you see in the figure above is therefore only valid for this specific case. Going back to the two researchers from our example, we can say that the HARKer has a better chance to come up with significant results than the Accumulator, if both do the same number of hypothesis tests.

Practice HARKing and Optional Stopping yourself

You can use the interactive p-hacker app to experience the efficiency of both p-hacking strategies yourself: You can increase the number of dependent variables and see whether one of them gets significant (HARing), or you can got to the “Now: p-hack!” tab and increase your sample until you obtain significance. Note that the DVs in the p-hacker app are not completely independent as in our example above, but rather correlate with r = .2, assuming that the DVs to some extent measure at least related constructs.

Conclusion

To conclude, we have shown how two p-hacking techniques work and why their application is bad for science. We found out that p-hacking techniques based on multiple testing typically end up with higher rates of false positive results than p-hacking techniques based on optional stopping, if we assume the same number of hypothesis tests. We want to stress that this does not mean that naive optional stopping is okay (or even okay-ish) in frequentist statistics, even if it does have a certain appeal. For those who want to do guilt-free optional stopping, there are ways to control for the \alpha error accumulation in the frequentist framework (see for example Wald, 1945, Chow & Chang, 2008, Lakens, 2014) and sequential Bayesian hypothesis tests (see for example our paper on sequential hypothesis testing with Bayes factors or Rouder, 2014).

Alternative Simulation Code (also including one-sided tests and t-tests)

sim.optstopping <- function(n.min, n.max, step, alpha=0.05, test="z.test", alternative="two.sided", iter=10000){

match.arg(test, choices=c("t.test", "z.test"))
match.arg(alternative, choices=c("two.sided", "directional"))

# Determine places of peek
peeks <- seq(n.min, n.max, by=step)

# Initialize result matrix (non-sequential)
res <- matrix(NA, ncol=length(peeks), nrow=iter)
colnames(res) <- peeks

# Conduct sequential testing (always until n.max, with peeks at pre-determined places)
for(i in 1:iter){
sample <- rnorm(n.max, 0, 1)
if(test=="t.test"){res[i,] <- sapply(peeks, FUN=function(x){mean(sample[1:x])/sd(sample[1:x])*sqrt(x)})}
if(test=="z.test"){res[i,] <- sapply(peeks, FUN=function(x){sum(sample[1:x])/sqrt(x)})}
}

# Create matrix: Which tests are significant?
if(test=="z.test"){
ifelse(alternative=="two.sided", signif <- abs(res) >= qnorm(1-alpha), signif <- res <= qnorm(alpha))
}else if (test=="t.test"){
n <- matrix(rep(peeks, iter), nrow=iter, byrow=T)
ifelse(alternative=="two.sided", signif <- abs(res) >= qt(1-alpha, df=n-1), signif <- res <= qt(alpha, df=n-1))
}

# Create matrix: Has there been at least one significant test in the trial?
seq.signif <- matrix(NA, ncol=length(peeks), nrow=iter)

for(i in 1:iter){
for(j in 1:ncol(signif)){
seq.signif[i,j] <- TRUE %in% signif[i, 1:j]
}
}

# Determine the sequential alpha error probability for the sequential tests
seq.alpha <- apply(seq.signif, MARGIN = 2, function(x){sum(x)/iter})

# Return a list of individual test p-values, sequential significance and sequential alpha error probability
return(list(p.values = res,
seq.significance = seq.signif,
seq.alpha = seq.alpha))

}
| Trackback

Two meanings of priors, part II: Quantifying uncertainty about model parameters

by Angelika Stefan & Felix Schönbrodt

This is the second part of “Two meanings of priors”. The first part explained a first meaning – “priors as subjective probabilities of models”. While the first meaning of priors refers to a global appraisal of existing hypotheses, the second meaning of priors refers to specific assumptions which are needed in the process of hypothesis building. The two kinds of priors have in common that they are both specified before concrete data are available. However, as it will hopefully become evident from the following blog post, they differ significantly from each other and should be distinguished clearly during data analysis.

The second meaning of priors: Prior distribution for parameters

In order to know how well evidence supports a hypothesis compared to another hypothesis, one must know the concrete specifications of each hypothesis. For example, in the tea tasting experiment, each hypothesis was characterized by a specific probability (e.g., the success rate of exactly 0.5 in HFisher of the previous blog post). What might sound trivial at first – deciding on the concrete specifications of a hypothesis – is in fact one of the major challenges when doing Bayesian statistics. Scientific theories are often imprecise, resulting in more than one plausible way to derive a hypothesis. With deciding upon one specific hypothesis, often new auxiliary assumptions are made. These assumptions, which are needed in order to specify a hypothesis adequately, are called “priors” as well. They influence the formulation and interpretation of the likelihood (which gives you the plausibility of data under a specific hypothesis). We will illustrate this in an example.

How much money do we spend for lunch?

Fig. 2: A typical lunch in Germany: Schweinshaxe, Brezn and 0.5 liter of beer.

A food company conducts market research in a large German city. They know from a recent representative enquiry by the German Federal Statistical Office that Germans spend on average 4.50 € for their lunch (standard deviation: 0.60 €). Now they want to know if the inhabitants of one specific city spend more money for their lunch compared to the German average. They expect lunch expenses to be especially high in this city because of the generally high living costs. In a traditional testing procedure in inferential statistics the food company would formulate two hypotheses to test their assumption: a null and an alternative hypothesis: H0: µ ≤ 4.50 and H1: µ > 4.50.

In Bayesian hypothesis testing, the formulation of the hypotheses has to be more precise than this. We need precise hypotheses as a basis for the likelihood functions which assign probability values to possible states of reality. The traditional formulation, µ > 4.50, is too vague for that purpose: Is any lunch cost above 4.50€ a priori equally likely? Is it plausible that a lunch costs 1,000,000€ on average? Probably not. Not every state of reality is, a priori, equally plausible. “Models connect theory to data“ (Rouder, Morey, & Wagenmakers, 2016), and a model that predicts everything predicts nothing.

As Bayesian statisticians we therefore must ask ourselves: Which values are more plausible given that our hypotheses are true? Of course, our knowledge differs from case to case in this point. Sometimes, we may be able to commit to a very small range of plausible values or even to a single value (in this case, we would call the respecting hypothesis a “point hypothesis”). Theories in physics sometimes predict a single state of reality: “If this theory is true, then the mass of a Chicks boson is exactly 1.52324E-16 gr”.

 

Point hypotheses in the food company example

More often, however, our knowledge about plausible values under a certain theory might be less precise, leading to a wider range of plausible values. Hence, the prior in the second sense defines the probability of a parameter value given a hypothesis, p(θ | H1).

Let us come back to the food company example. Their null hypothesis might be that there is no difference between the city in the focus of their research project and the German average. Hence, the null hypothesis predicts an average lunch cost of 4.50€. With the alternative hypothesis, it becomes slightly more complex. They assume that average lunch expenses in the city should be higher than the German average, so the most plausible value under the alternative hypothesis should be higher than 4.5. However, they may deem it very improbable that the mean lunch expenses are more than two standard deviations higher than the German average (so, for example, it should be very improbable that someone spends more than, say, 10 EUR for lunch even in the expensive city). With this knowledge, they can put most plausibility on values in a range from 4.5 to 5.7 (4.5 + 2 standard deviations). They could further specify their hypothesis by claiming that the most plausible value should be 5.1, i.e., one standard deviation higher than the German average. The elements of these verbal descriptions of the alternative hypothesis can be summarized in a truncated normal distribution that is centered over 5.1 and truncated at 4.5 (as the directional hypothesis does not predict values in the opposite direction).

 

Truncated normal distribution centered over 5.1 and truncated at 4.5

With this model specification, the researchers would place 13% of the probability mass on values larger than 2SD of the general population (i.e., > 5.7).

Making it even more complex, they could quantify their uncertainty about the most plausible value (i.e., the maximum of the density distribution) by assigning another distribution to it. For example, they could build a normal distribution around it, with a mean of 5.1 and a standard deviation of 0.3. This would imply that in their opinion, 5.1 is the “most plausible most plausible value” but that values between 4.8 and 5.4 are also potential candidates for the most plausible value.

Why making it complicated if it’s so simple in traditional hypothesis testing?

What you can notice in the example about the development of hypotheses is that the market researchers have to make auxiliary assumptions on top of their original hypothesis (which was H1: µ > 4.5). If possible, these prior plausibilities should be informed by theory or by previous empirical data. Specifying alternative hypothesis in this way may seem to be an unnecessary burden compared to traditional hypothesis testing where these extra assumptions seemingly are not necessary. Except that they are necessary. Without going into detail in this blog post, we recommend to read Rouder et al.’s (2016a) “Is there a free lunch in inference?“, with the bottom line that principled and rational inference needs specified alternative hypotheses. (For example, in Neyman-Pearson testing, you also need to specify a precise alternative hypothesis that refers to the “smallest effect size of interest”)

Furthermore, readers might object: “Researchers rarely have enough background knowledge to specify models that predict data“. Rouder et al. (2016b) argue that this critique is overstated, as (1) with proper elicitation, researchers often know much more than they initially think, (2) default models can be a starting point if really no information is available, and (3) several models can be explored without penalty.

Do the two kinds of priors depend on each other in some way?

A question that may come to your mind soon after you understood the difference between the two kinds of priors is: If they both are called “priors”, do they depend on each other in some way? Does the formulation of your “personal prior plausibility of a hypothesis” (like the skeptical observer’s prior on Mrs. Bristol’s tea tasting abilities) influence the specification of your model (like the hypothesis specification in the second example) or vice versa?

The straightforward answer to this question is “no, they don’t”. This can be easily illustrated in a case where the prior conviction of a researcher runs against the hypothesis he or she wants to test. The food company in the second example has sophisticatedly determined the likelihood of the two hypotheses (H0 and H1), which they want to pit against each other. They are probably considerably convinced that the specification of the alternative hypothesis describes reality better than the specification of the null hypothesis. In a simplified form, their prior odds (i.e., priors in the first sense) can be described as a ratio like 10:1. This would mean that they deem the alternative hypothesis ten times as likely as the null hypothesis. However, another food company, may have prior odds of 3:5 while conducting the same test (i.e., using the same prior plausibilities of model parameters). This shows that priors in the first sense are independent of priors in the second sense. Priors in the first sense change with different personal convictions while priors in the second sense remain constant. Similarly, prior beliefs can change after seeing the data – the formulation of the model (i.e., what a theory predicts) stays the same. (As long as the theory, from which the model specification is derived, does not change. In an estimation context, the model parameters are updated by the data.)

Summary of both blog posts

The term “prior” has two meanings in the context of Bayesian hypothesis testing. The first one, usually applied in Bayes factor analysis, is equivalent to a prior subjective probability of a hypothesis (“how plausible do you deem a hypothesis compared to another hypothesis before seeing the data”). The second meaning refers to the assumptions made in the specification of the model of the hypotheses which are needed to derive the likelihood function. These two meanings of the term “prior” have to be distinguished clearly during data analysis, especially as they do not depend on each other in any way. Some researchers (e.g., Dienes, 2016) therefore suggest to call only priors in the first sense “priors” and speak about “specification of the model” when referring to the second meaning.

Read the first part of this blog post: Priors as the plausibility of models

References

Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on?. Perspectives On Psychological Science, 6(3), 274-290. http://doi:10.1177/1745691611406920

Dienes, Z. (2016). How Bayes factors change scientific practice. Journal Of Mathematical Psychology, 7278-89. http://doi:10.1016/j.jmp.2015.10.003

Lindley, D. V. (1993). The analysis of experimental data: The appreciation of tea and wine. Teaching Statistics, 15(1), 22-25. http://dx.doi.org/10.1111/j.1467-9639.1993.tb00252.x

Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016a). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520–547. http://doi.org/10.1111/tops.12214

Rouder, J. N., Morey, R. D., & Wagenmakers, E. J. (2016b). The Interplay between Subjectivity, Statistical Practice, and Psychological Science. Collabra, 2(1), 6–12. http://doi.org/10.1525/collabra.28

library(truncnorm)

# parameters for H1 model
M <- 5.1
SD <- 0.5

range <- seq(4.5, 7, by=.01)
plausibility <- dtruncnorm(range, a=4.5, b=Inf, mean=M, sd=SD)
plot(range, plausibility, type="l", xlim=c(4, 7), axes=FALSE, ylab="Plausibility", xlab="Lunch cost in €", mgp = c(2.5, 1, 0))
axis(1)

# Get the axis ranges, draw arrow
u <- par("usr")
points(u[1], u[4], pch=17, xpd = TRUE)
lines(c(u[1], u[1]), c(u[3], u[4]), xpd = TRUE)

abline(v=4.5, lty="dotted")

# What is the probability of values > 5.7?
1-ptruncnorm(5.7, a=4.5, b=Inf, mean=M, sd=SD)
Comments (1) | Trackback

Two meanings of priors, part I: The plausibility of models

by Angelika Stefan & Felix Schönbrodt

When reading about Bayesian statistics, you regularly come across terms like “objective priors“, “prior odds”, “prior distribution”, and “normal prior”. However, it may not be intuitively clear that the meaning of “prior” differs in these terms. In fact, there are two meanings of “prior” in the context of Bayesian statistics: (a) prior plausibilities of models, and (b) the quantification of uncertainty about model parameters. As this often leads to confusion for novices in Bayesian statistics, we want to explain these two meanings of priors in the next two blog posts*. The current blog post covers the the first meaning of priors (link to part II).

The first meaning of “prior”: Prior plausibility for models

In this context, the term “prior” incorporates the personal assumptions of a researcher on the probability of a hypothesis (p(H1)) relative to a competing hypothesis, which has the probability p(H2). Hence, the meaning of this prior is “how plausible do you deem a model relative to another model before looking at your data”. The ratio of the two priors of the models, that is “how probable do you consider H1 compared to H2”, is called “prior odds”: p(H1) / p(H2).

The first meaning of priors is used in the context of Bayes factor analysis, where you compare two different hypotheses. In Bayes factor analysis, prior odds are updated by the likelihood ratio of the two hypotheses, which contains the information from the data, and result in the posterior odds (“what you believe after looking at your data”):

\large \frac{Posterior_{H1}}{Posterior_{H2}} =\frac{Likelihood_{H1}}{Likelihood_{H2}}\cdot\frac{Prior_{H1}}{Prior_{H2}}

The prior belief is called “subjective”, but this label does not imply that it is “arbitrary”, “unprincipled”, or “irrational”. In contrast, the prior belief can (and preferably should) be informed by previous data or experiences. For example, it can be a belief that started with an equipoise (50/50) position, but has been repeatedly updated by data. But within the bounds of rationality and consistency, people still can come to considerably different prior beliefs, and might have good arguments for their choice – that’s why it is called “subjective”. But initially differing degrees of belief will converge as more and more evidence comes in. We will observe this in the following example.

The incredible tea tasting abilities of Lady Muriel Bristol

Fig. 1: Muriel Bristol surprising her husband (William) by making the correct guess for the fifth time in a row (reenacted scene)

The classical experiment of tasting tea has already been described in the context of Bayesian hypothesis testing by Lindley (1993). We will present a simpler form here. Dr. Muriel Bristol, a scientist working in the field of alga biology who was acquainted to the statistician R. A. Fisher, claimed that she could discriminate whether milk is put in a cup before or after the tea infusion during the process of preparing tea with milk. However, Mr. Fisher considered this very unlikely.

So they decided to run an experiment: Muriel Bristol tasted several cups of tea in a row, making a guess on the preparation procedure for each cup. Unlike in the original story, where inferential statistics were consulted to solve the disagreement, we will employ Bayesian statistics to track how prior convictions in this example change. If Muriel Bristol makes her guesses only based on chance as Mr. Fisher supposes, she has a probability of success of 50% in each trial. Before observing her performance, Mr. Fisher should therefore consider it very likely that she is right about the procedure in about 50% of the trials across all trials. We can therefore assume a point hypothesis: HFisher: Success rate (SR) = 0.5. Muriel Bristol, on the other hand, is very confident in her divination skills. She claims to get 80% of trials correct, which can be equally captured in a point hypothesis: HMuriel: SR = 0.8.

Two observers with different prior beliefs

To introduce prior beliefs about hypotheses and show how they change with upcoming evidence, we want to introduce two additional persons. The first one is a slightly skeptical observer who tends to favor HFisher, but does not completely rule out that Mrs. Bristol could be right with her hypothesis. More formally, we could describe this position as: P(HFisher) = 0.6 and P(HMuriel) = 0.4. This means that his prior odds are P(HFisher)/P(HMuriel) = 3:2. Fisher’s hypothesis is 1.5 times more likely to him than Muriel Bristol’s hypothesis.

The second additional person we would like to introduce is William, Muriel Bristol’s loving husband who fervently advocates her position. He knows his wife would never (or at least very rarely) make wrong claims, concerning tea preparation and all others issues of their marriage. He therefore assigns a much higher subjective probability to her hypothesis (P(HMuriel) = 0.9) than to the one of Mr. Fisher (P(HFisher) = 0.1). His prior odds are therefore P(HFisher)/P(HMuriel) = 1:9. Please note that the content of the hypotheses (the proposed success rates 0.5 and 0.8, which are the parameters of the model) is logically independent of the probability of the hypotheses (priors) that our two observers have.

How to update prior beliefs about hypotheses

During the process of hypothesis testing, these two priors are updated with the existing evidence. It is reported that Muriel Bristol’s performance at the experiment was extraordinarily good. We therefore assume that out of the first 6 trials of the experiment she got 5 correct.

With this information, we can now compute the likelihood of the data under each of the hypotheses (for more information on the computation of likelihoods, see Alexander Etz’s blog:

L_{Fisher} = 0.5^5 \cdot 0.5^1 =  0.016

L_{Muriel} = 0.8^5 \cdot 0.2^1 = 0.066

The computation of the likelihoods does not involve the prior model probabilities of our observers. What can be seen is that the data are more likely under Muriel Bristol’s hypothesis than under Mr. Fisher’s. This should not come as a surprise as Muriel Bristol claimed that she could make a very high percentage of right guesses and the data show a very high percentage of right guesses whereas Mr. Fisher assumed a much lower percentage of right guesses. To emphasize this difference in likelihoods and to assign it a numerical value, we can compute the likelihood ratio (Bayes factor):

BF_{FM} = LR_{FM} = \frac{L_{Fisher}}{L_{Muriel}} = 0.016 / 0.066 = 0.238

BF_{MF} = LR_{MF} = \frac{L_{Muriel}}{L_{Fisher}} = 0.066 / 0.016 = 4.19

This ratio means that the data are 4.19 times as likely under Mrs. Bristol’s hypothesis as under Mr. Fisher’s hypothesis. It does not matter how you order the likelihoods in the fraction, the meaning remains constant (see this blog post).

How does this likelihood change the prior odds of both our slightly skeptical observer and William Bristol? Bayes theorem shows that prior odds can be updated by multiplying them with the likelihood ratio (Bayes factor):

\frac{ Posterior(H_{Fisher}) } {Posterior(H_{Muriel})} =  \frac{Prior(H_{Fisher})}{Prior(H_{Muriel})} \cdot \text {Bayes Factor}

First, we will focus on the posterior odds of the slightly skeptical observer. To remember, the slightly skeptical observer had assigned a probability of 0.6 to Mr. Fisher’s hypothesis and a probability of 0.4 to Muriel Bristol’s hypothesis before seeing the data, which resulted in prior odds of 3:2 for Mr. Fisher’s hypothesis. How do these convictions change now when the experiment has conducted? To examine this, we simply have to insert all known values in the equation:

\frac {3}{2} \cdot 0.238 = 0.357 = \frac {1}{2.8}

This shows that the prior odds of the slightly skeptical observer changed from 3:2 to posterior odds of 1:2.8. This means that whereas before the experiment the slightly skeptical observer had deemed Mr. Fisher’s hypothesis more plausible than Mrs. Bristol’s hypothesis, he changed his opinion after seeing the data, now preferring Mrs. Bristol’s hypothesis over Mr. Fisher’s.

The same equation can be applied to William Bristol’s prior odds:

\frac {1}{9} \cdot 0.238 = 0.0264 = \frac {1}{37.9}

What we can notice is that after taking the data into consideration both prior odds display a higher amount of agreement with Muriel Bristol and reduced confidence in Mr. Fisher’s hypothesis. Whereas the convictions of the slightly skeptical observer were changed in favor of Muriel Bristol’s hypothesis after the experiment, William Bristol’s prior convictions were strengthened.

Something else you can notice is that compared to William Bristol the slightly skeptical observer still assigns a higher plausibility to Mr. Fisher’s hypothesis. This rank order between the two priors will remain no matter what the data look like. Even if Muriel Bristol made, say, 100/100 correct guesses, the slightly skeptical observer would trust less in her hypothesis than her husband. However, with increasing evidence the absolute difference between both observers will decrease more and more.

Summary

This blog post explained the first meaning of “prior” in the context of Bayesian statistics. It can be defined as the subjective plausibility a researcher assigns to a hypothesis compared to another hypothesis before seeing the data. As illustrated in the tea-tasting example, these prior beliefs are updated with upcoming evidence in the research process. In the next blog post, we will explain a second meaning of “priors”: The quantification of uncertainty about model parameters.

Continue reading part II: Quantifying uncertainty about model parameters

We want to thank Eric-Jan Wagenmakers for helpful comments on a previous version of the post.

*As a note: Both meanings in fact can be unified, but for didactic purpose we think it makes sense to keep them distinct as a start.

References

Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on?. Perspectives On Psychological Science, 6(3), 274-290. http://doi:10.1177/1745691611406920

Dienes, Z. (2016). How Bayes factors change scientific practice. Journal Of Mathematical Psychology, 7278-89. http://doi:10.1016/j.jmp.2015.10.003

Lindley, D. V. (1993). The analysis of experimental data: The appreciation of tea and wine. Teaching Statistics, 15(1), 22-25. http://dx.doi.org/10.1111/j.1467-9639.1993.tb00252.x

Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016a). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520–547. http://doi.org/10.1111/tops.12214

Rouder, J. N., Morey, R. D., & Wagenmakers, E. J. (2016b). The Interplay between Subjectivity, Statistical Practice, and Psychological Science. Collabra, 2(1), 6–12. http://doi.org/10.1525/collabra.28

Comments (4) | Trackback
Send this to a friend