[Update 2015/1/14: I consolidate feedback from Twitter, comments, email, and real life into the main text (StackExchange-style), so that we get a good and improving answer. Thanks to @TonyLFreitas, @PhDefunct, @bahniks, @JoeHilgard, @_r_c_a, @richardmorey, @R__INDEX, the commenters at the end of this post and on the OSF mailing list, and many others for their feedback!]
In a recent lecture I talked about the replication crisis in psychology. After the lecture my students asked: “We learn so much stuff in our lectures, and now you tell us that a considerable proportion of these ‘facts’ probably are just false positives, or highly exaggerated? Then, what can we believe at all?”. A short discussion soon led to the crucial question:
In the era of #repligate: What are valid cues for the trustworthiness of a study?
Valid cues for trustworthiness of a single study:
- Pre-registration. This might be one of the strongest cues for trustworthiness. Pre-registration makes p-hacking and HARKing unlikely (Wagenmakers, Wetzels, Borsboom, Maas, & Kievit, 2012), and takes care for a sufficient amount of statistical power (At least, some sort of sample size planning has been done. Of course, this depends on the correctness of the a-priori effect size estimate).
- Sample size / Statistical Power. Larger samples mean higher power, higher precision, and less false positives (Bakker, van Dijk, & Wicherts, 2012; Maxwell, Kelley, & Rausch, 2008; Schönbrodt & Perugini, 2013). Of course sample size alone is not a panacea. As always, the garbage in/garbage out principle holds, and a well designed lab study with n=40 can be much more trustworthy than a sloppy mTurk study with n=800. But all other things being equal, I put more trust in larger studies.
- Independent high-power replications. If a study has been independently replicated from another lab with high power and preferably pre-registered, this probably is the strongest evidence for the trustworthiness of a study (How to conduct a replication? See the Replication Recipe by Brandt et al., 2014).
- I guess that studies with Open Data and Open Material have a higher replication rate
- “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results” (Wicherts, Bakker, & Molenaar, 2011) —> this is not exactly Open Data, because here authors only shared data upon request (or not). But it points into the same direction.
- Beyond publishing Open Data at all, the neatness of the data set and the quality of the analysis script is an indicator (see also comment by Richard Morey). The journal “Quarterly Journal of Political Science” demands to publish raw data and analysis code that generates all the results reported in the paper. Of these submissions, 54% “had results in the paper that differed from those generated by the author’s own code”! My fear is that analytical code that has not been refined and polished for publishing contains even more errors (not to speak of unreproducible point-and-click analyses). Therefore, a well prepared data set and analysis code should be a valid indicator.
- Open Material could be an indicator that people are not afraid of replications and further scrutiny
- An abstract with reasonable conclusions that stick close to the data – see also below: “Red flags”. This includes visible efforts of the authors to explain how they could be wrong and what precautions were/were not taken.
- A sensitivity analysis, which shows that conclusions do not depend on specific analytical choices. For Bayesian analyses this means to explore how the conclusions depend on the choice of the prior. But you could also show how your results change when you do not exlude the outliers, or do not apply that debatable transformation to your data (see also comment)
- Using the “21 Word Solution” of Simmons, Nelson, & Simonsohn (2012) leads to a better replication index.
Valid cues for trustworthiness of a research programme/ multiple studies:
Valid cues for UNtrustworthiness of a single study/ red flags:
In a comment below, Dr. R introduced the idea of “red flags”, which I really like. These red flags aren’t a prove of the untrustworthiness of a study – but definitely a warning sign to look closer and to be more sceptical.
- Sweeping claims, counterintuitive, and shocking results (that don’t connect to the actual data)
- Most p values are in the range of .03 – .05 (or, equivalently, most t-values in the 2-3 range, or most F-values are in the 4-9 range; see comment by Dr. R below).
- How does a distribution of p values look like when there’s an effect? See Daniël Lakens blog. With large samples, p-values just below .05 even indicate support for the null!
- It’s a highly cited result, but no direct replications have been published so far. That could be an indicator that many unsuccessful replication attempts went into the file-drawer (see comment by Ruben below).
- Too good to be true: If several low-power studies are combined in a paper, it can be very unlikely that all of them produce significant results. The “Test of Excess Significance” has been used to formally test for “too many significant results”. Although this formal test has been criticized (e.g., see The Etz-Files, and especially the long thread of comments, or this special issue on the test), I still think excess significance can be used as a red flag indicator to look closer.
Possibly invalid cues (cues which are often used, but only seemingly are indicators for a study’s trustworthiness):
- The journal’s impact factor. Impact factors correlate with retractions (Fang & Casadevall, 2011), but do not correlate with a single paper’s citation count (see here).
- I’m not really sure whether that is a valid or invalid cue for a study’s quality. The higher retraction rate might due to the stronger public interest and a tougher post-publication review of papers in high-impact journals. The IF seems not to be predictive of a single paper’s citation count; but I’m not sure either whether the citation count is an index of a study’s quality. Furthermore, “Impact factors should have no place in grant-giving, tenure or appointment committees.” (ibid.), see also a reccent article by @deevybee in Times Higher Education.
- On the other hand, the current replicability estimate of a full volume of JPSP is only at 20-30% (see Reproducibility Project: Psychology). A weak performance for one of our “best journals”.
- The author’s publication record in high-impact journals or h-index. This might be a less valid cue as expected, or even an invalid cue.
- Meta-analyses. Garbage-in, garbage-out: Meta-analyses of a biased literature produce biased results. Typical correction methods do not work well. When looking at meta-analyses, at least one has to check whether and how it was corrected for publication bias.
“If a study has a large sample size, Open Data, and maybe even has been pre-registered, I would put quite some trust into the results. If the study has been independently replicated, even better. In contrast to common practice, I do not care so much whether this paper has been published in a high-impact journal or whether the author has a long publication record. The next step, of course, is: Read the paper, and judge it’s validity and the quality of its arguments!”
This list certainly is not complete, and I would be interested in your ideas, additions, and links to relevant literature!
Recently, a student of mine (Felix Süßenbach, now at the University of Edinburgh) and I published a little study on gaze-cueing, and how it is moderated by the trustworthiness of the gazing person.
In a nutshell, although instructed to ignore the gaze, participants shifted their attention into the direction where another person looked at (–> this is the well-established gaze-cueing effect), but more so when the sender was introduced as being trustworthy (–> which is the new result)
We also found some exploratory evidence that the trait anxiety of participants moderates that effect, in a way that highly anxious participants did not differentiate between trustworthy and untrustworthy senders: Highly anxious participants always followed the other person’s gaze. For low anxious participants, in contrast, the gaze-cueing effect was reduced to zero for untrustworthy senders. (This exploratory finding, of course, awaits cross-validation).
The paper, raw data, and R script for the analyses are on OSF.
Süßenbach, F., & Schönbrodt, F. (2014). Not afraid to trust you: Trustworthiness moderates gaze cueing but not in highly anxious participants. Journal of Cognitive Psychology, 26, 670–678. doi:10.1080/20445911.2014.945457
Publisher’s website: http://www.tandfonline.com/doi/abs/10.1080/20445911.2014.945457#.VEn3GVu17sI
Abstract Gaze cueing (i.e., the shifting of person B’s attention by following person A’s gaze) is closely linked with human interaction and learning. To make the most of this connection, researchers need to investigate possible moderators enhancing or reducing the extent of this attentional shifting. In this study we used a gaze cueing paradigm to demonstrate that the perceived trustworthiness of a cueing person constitutes such a moderator for female participants. Our results show a significant interaction between perceived trustworthiness and the response time trade-off between valid and invalid gaze cues [gaze cueing effect (GCE)], as manifested in greater following of a person’s gaze if this person was trustworthy as opposed to the following of an untrustworthy person’s gaze. An additional exploratory analysis showed potentially moderating influences of trait-anxiety on this interaction (p = .057). The affective background of the experiment (i.e., using positive or negative target stimuli) had no influence.
This is a post-publication peer review (HIBAR: “Had I Been A Reviewer”) of the following paper:
Levi, M., Li, K., & Zhang, F. (2010). Deal or no deal: Hormones and the mergers and acquisitions game. Management Science 56, 1462 -1483.
A citeable version of this post-publication peer review can be found at SSRN:
In their article “Deal or No Deal: Hormones and the Mergers and Acquisitions Game,” Levi, Li, and Zhang (2010) claimed that they investigated the effect of testosterone on CEOs’ decisions in mergers and acquisitions. However, they did not measure testosterone levels directly. Rather, they tried to use CEO age as a proxy, based on a previously documented negative correlation between age and testosterone level. In this comment, I argue that it is not reasonable to use age as a proxy for testosterone, and that Levi et al.’s study does not tell us anything about testosterone. General remarks on using proxy variables are given.
In their article “Deal or No Deal: Hormones and the Mergers and Acquisitions Game,” Levi, Li, and Zhang (2010) investigated the research question of whether the hormone testosterone (T) has an impact on decisions in mergers and acquisitions (M&As). Based on experimental results that T has an effect on behavior in ultimatum games (Burnham, 2007), Levi et al. hypothesized that CEOs with higher T levels should show more aggressive/dominant behavior in M&As. To investigate this hypothesis, the authors assembled a data set with 357 M&As and several economic variables related to them (e.g., the size of the target firm, the board size, and several other economic indicators). As they could not assess the T levels of the CEOs directly, they “[…] have suggested an alternative: specifically, to proxy testosterone by age” (p. 1476). Therefore, as the authors admitted themselves, their reasoning was based on a central assumption: “The validity of this approach clearly depends on the extent of the association of hormone levels with age.” (p. 1476). To summarize their findings, a significant but small effect of age on M&A decisions was found: younger CEOs made more bid withdrawals and more tender offers than their older counterparts (which has been interpreted as more dominant behavior). As the story has received wide press coverage, for example, in the Wall Street Journal, Financial Times, Time Magazine, and the Los Angeles Times, I feel the need to make some clarifications.
Empirically, Levi et al. have shown an effect of CEO age on the outcome of M&As. From the title to the conclusion of their article, however, they refer to the effect of testosterone (e.g., “[…] in M&As the testosterone of both parties could influence the course and outcome of negotiations,” p. 1463; “[…] we consider whether testosterone influences the likelihood that offers made are subsequently withdrawn,” p.1466; “This finding strongly supports an association between testosterone, as proxied by the bidder male CEO age, and M&As,” p. 1469).
In the following commentary, I argue that it is not appropriate to use age as a proxy for T level and that the conclusions of Levi et al. are taking it way too far. For the clarity of my arguments, I will focus only on the strongest reported effect. For all weaker effects, the same reasoning applies even more.
The Effect of Testosterone on Dominant Behavior is Rather Low
Is it a reasonable hypothesis to expect more dominant M&A behavior from CEOs with higher T levels? Early investigations with animals have shown a relation between testosterone level and aggressive or dominant behavior (e.g., Wingfield, Hegner, Dufty, & Ball, 1990). Recent review articles and meta-analyses on human testosterone, however, have shown that the effects in humans are rather small. For example, in a meta-analysis on the relation of male human aggression and T level (N = 9760; Archer, Graham-Kevan, & Davies, 2005; Book, Starzyk, & Quinsey, 2001), the weighted correlation was only .08. Other meta-analyses on the relation of T to dominance (weighted = .13) and to a challenge in a sports competition (weighted = .18) have supported this finding of a small effect (for an overview, see Archer, 2006). To summarize, if M&As are seen as competitive situations, indeed, an effect of T could be expected – but the relation is probably much less pronounced than common sense might suggest.
The Relation Between Age and Testosterone is Rather Low
The central assumption in their article is that T level can be proxied by age. How strong is that relation? A meta-analysis of 23 studies reporting the correlation between age and T level (N = 1900) revealed that the average correlation between these variables was -.18 (Gray, Berlin, McKinlay, & Longcope, 1991). That means only 3% of the variation in T level can be explained by age. Levi et al. refer to another study that shows a remarkably stronger correlation of -.50 (Harman, Metter, Tobin, Pearson, & Blackman, 2001).
All articles cited by Levi et al. report the correlation for a broad age range (e.g., 24 – 90 years, Ferrini & Barrett-Conner, 1998; 23 – 91 years, Harman et al., 2001). Given that 90% of CEOs in Levi et al.’s sample were between 46 and 64, a massive range restriction is present, which presumably lowers the correlation in that age range even more. Indeed, the scatter plots in Harman et al. (2001) strongly suggest that the correlation is mainly driven by the very young and very old participants.
But regardless of whether the true correlation is closer to -.18 or to -.50: Are these correlations high enough that age is warranted as a valid proxy for T level? As we will see in the next section, the answer is no.
A “Triangulation” of an Unobserved Correlation?
In their conclusions, the authors argue as though they had established a causal effect of T on M&A decisions. However, they had not even established a correlation between T and M&A behavior. Their approach seems to be something like a “triangulation” of an unobserved correlation by the knowledge of two other correlations. Indeed, there are dependencies and constraints in the relation of the bivariate correlations between three variables. If two of the three correlations are given ( and ), the possible range of the third correlation, , is constrained by following equation (Olkin, 1981):
Figure 1 graphically represents the relationship of variables given by this equation. The left plot shows the upper boundary of , the right plot the lower boundary of . As one can see, rather high values for either , , or both, have to be present to imply a positive sign (i.e., a lower boundary > 0) of .
Given the observed estimates of = -.50 and = -.12, the range of possible values for goes from -.80 to +.92. (The position is marked by the asteriks in Figure 1.) If the probably more realistic estimate from the meta-analysis is used ( = -.18; Gray et al., 1991), the possible range for goes from -.95 to +1.00. These calculations clearly show that there is no argument to expect a positive correlation between T and M&A decisions. Actually, with the given data set, no conclusions about T can be drawn at all.
The Effect of Age on M&A Decisions: An Artefact?
Given these analyses, it should be clear that one cannot speak of a T effect based on these data. What about the age effect reported in this article? The authors wrote that, “[… m]otivated by the studies of population testosterone levels reviewed earlier, we use the age of 45 years as the cut-off to separate young male CEOs from the rest.” (p. 1467). This seems to be a problematic choice to me. First of all, none of the cited studies gives a hint about why the age of 45 should be a particularly meaningful cut point. None of the reviewed studies suggests a significant break or stronger decline of T at that age. Furthermore, that cut point leads to a very skewed distribution of 16 “young” vs. 341 “old” CEOs.
The strongest reported effect was that of the dichotomized variable “CEO is young” (i.e., younger than 45) on bid withdrawals. Based on the reported descriptive statistics and correlations, it can be computed that 5 young CEOs withdrew, whereas 11 did not. Concerning old CEOs, 40 out of 341 withdrew. This distribution led to the reported Pearson correlation of = .12 ( = .02). If only one young CEO would not have withdrawn, the correlation would have been nonsignificant ( = .08, = .13). If only three young CEOs would not have withdrawn, age and withdrawing behavior would be completely unrelated. Hence, as Levi et al.’s conclusions stand and fall on the decision of one single CEO, these results do not seem very robust to me. Why is age categorized at all? Methodological papers on the topic clearly suggest not to dichotomize if a predictor variable can be used on an interval or ratio scale (e.g., Royston, Altman, & Sauerbrei, 2006).
Referring to several meta-analyses, it could be shown that the relation between T and dominant human behavior is much less pronounced than suggested by Levi and colleagues. Likewise, the relation between age and testosterone, especially in the restricted age range of their sample, is presumably close to zero. From my reading of the empirical evidence, the article should be rewritten as “Age and the mergers and acquisitions game,” and it should be acknowledged that the age effect, although significant, has a small effect size.
Of course, age is not a psychological variable per se – it is always a proxy for a true causal factor: “And although it is unquestionably useful to find that a phenomenon covaries with age, neither age nor the related variable time is a causal variable; changes occur in time, but not as a result of time.” (Hartmann & George, 1999, p. 132). Hence, it would be feasible to spend one or two paragraphs in the discussion speculating about the potential causal role of testosterone (and several other age-related variables). In that article, however, from the main title to the conclusion, the authors argue that T is the causal factor. Although the authors try to rule out some alternative age-related factors, the main criticism remains: With the current data, there is no way to provide evidence for or against a T hypothesis. It’s simply the wrong story for the data.
Based on the same data set, several other articles could have been written in the same style, for example, “Bicep strength and the M&A game.” Why bicep strength? The theory of embodied cognition predicts that physical dominance is a predictor for psychological dominance in negotiations (Barsalou, 2008). As CEO muscle strength cannot be measured directly, one could use CEO age as a proxy for muscle strength ( =.55; Lindle et al., 1997). Physically stronger CEOs, as proxied by age, would therefore be expected to make more dominant choices in M&As. This ad hoc hypothesis has the same argumentative structure and the same empirical justification as the “testosterone story,” and the creative reader can think of a multitude of other age-related variables that might have an influence on negotiation decisions (e.g., generational cohort effects in moral values, fluid intelligence, different educational levels between age groups, time spent in marriage, or the grayishness of the CEO’s hair).
To summarize, Levi et al. conclude that they “[…] have been able to conduct that age appears to be proxying for testosterone rather than experience, horizon, or some other effect” (p. 1478) and that their finding “strongly supports an association between testosterone, as proxied by the bidder male CEO age, and M&As.” (p. 1469). In the light of the analyses given above, that conclusion is not justified.
Archer, J. 2006. Testosterone and human aggression: An evaluation of the challenge hypothesis. Neuroscience & Biobehavioral Reviews 30(3) 319-345.
Archer, J., N. Graham-Kevan, M. Davies. 2005. Testosterone and aggression: A reanalysis of Book, Starzyk, and Quinsey’s (2001) study. Aggression and Violent Behavior 10(2) 241-261.
Barsalou, L.W. 2008. Grounded cognition. Annu. Rev. Psychol. 59 617–645.
Book, A.S., K.B. Starzyk, V.L. Quinsey. 2001. The relationship between testosterone and aggression: A meta-analysis. Aggression and Violent Behavior 6(6) 579–599.
Burnham, T. C. 2007. High-testosterone men reject low ultimatum game offers. Proceedings of the Royal Society B: Biological Sciences 274(1623) 2327-2330.
Cohen, J. 1992. A power primer. Psychological Bulletin 112(1) 155-159.
Ferrini, R. L., E. Barrett-Connor. 1998. Sex hormones and age: a cross-sectional study of testosterone and estradiol and their bioavailable fractions in community-dwelling men. American Journal of Epidemiology 147 750-754.
Gray, A., J. A. Berlin, J. B. McKinlay, C. Longcope. 1991. An examination of research design effects on the association of testosterone and male aging: results of a meta-analysis. Journal of Clinical Epidemiology 44(7) 671-684.
Harman, S. M., E. J. Metter, J. D. Tobin, J. Pearson, M. R. Blackman. 2001. Longitudinal effects of aging on serum total and tree testosterone levels in healthy men. Journal of Clinical Endocrinology & Metabolism 86(2) 724 -731.
Hartmann, D. P., T. P. George. 1999. Design, measurement, and analysis in developmental research. M. H. Bornstein, M. E. Lamb, ed. Developmental psychology: An advanced textbook (4th ed.). Mahwah, NJ US: Lawrence Erlbaum Associates Publishers, 125-195.
Levi, M., K. Li, F. Zhang. 2010. Deal or no deal: Hormones and the mergers and acquisitions game. Management Science 56(9) 1462 -1483.
Lindle, R. S., E. J. Metter, N. A. Lynch, J. L. Fleg, J. L. Fozard, J. Tobin, T. A. Roy, B. F. Hurley. 1997. Age and gender comparisons of muscle strength in 654 women and men aged 20–93 yr. Journal of Applied Physiology 83(5) 1581-1587.
Olkin, I. 1981. Range restrictions for product-moment correlation matrices. Psychometrika 46(4) 469-472.
Royston, P., D. G. Altman, W. Sauerbrei. 2006. Dichotomizing continuous predictors in multiple regression: A bad idea. Statistics in Medicine 25(1) 127-141.
Wingfield, J. C., R. E. Hegner, A. M. Dufty, G. F. Ball. 1990. The ‘Challenge Hypothesis’: Theoretical Implications for Patterns of Testosterone Secretion, Mating Systems, and Breeding Strategies. The American Naturalist 136(6) 829-846.
Reanalyzing the Schnall/Johnson “cleanliness” data sets: New insights from Bayesian and robust approaches
I want to present a re-analysis of the raw data from two studies that investigated whether physical cleanliness reduces the severity of moral judgments – from the original study (n = 40; Schnall, Benton, & Harvey, 2008), and from a direct replication (n = 208, Johnson, Cheung, & Donnellan, 2014). Both data sets are provided on the Open Science Framework. All of my analyses are based on the composite measure as dependent variable.
This re-analsis follows previous analyses by Tal Yarkoni, Yoel Inbar, and R. Chris Fraley, and is focused on one question: What can we learn from the data when we apply modern (i.e., Bayesian and robust) statistical approaches?
The complete and reproducible R code for these analyses is at the end of the post.
Disclaimer 1: This analysis assumes that the studies from which data came from were internally valid. Of course the garbage-in-garbage-out principle holds. But as the original author reviewed the experimental material of the replication study and gave her OK, I assume that the relication data is as valid as the original data.
Disclaimer 2: I am not going to talk about tone, civility, or bullying here. Although these are important issues, a lot has been already written about it, including apologies from one side of the debate (not from the other, yet), etc. For nice overviews over the debate, see for example a blog post by Tal Yarkoni). I am completely unemotional about these data. False positives do exists, I am sure I had my share of them, and replication is a key element of science. I do not suspect anybody of anything – I just look at the data., and the summary provided by
That being said, let’s get to business:
Bayes factor analysis
The BF is a continuous measure of evidence for H0 or for H1, and quantifies, “how much more likely is H1 compared to H0″. Typically, a BF of at least 3 is requested to speak of evidence (i.e., the H1 should be at least 3 times more likely than the H0 to speak of evidence for an effect). For an introduction to Bayes factors see here, here, or here.
Using the BayesFactor package, it is simple to compute a Bayes factor (BF) for the group comparison. In the original study, the Bayes factor against the H0, , is 1.08. That means, data are 1.08 times more likely under the H1 (“there is an effect”) than under the H0 (“There is no effect”). As the BF is virtually 1, data occurred equally likely under both hypotheses.
What researchers actually are interested in is not p(Data | Hypothesis), but rather p(Hypothesis | Data). Using Bayes’ theorem, the former can be transformed into the latter by assuming prior probabilities for the hypotheses. The BF then tells one how to update one’s prior probabilities after having seen the data. Given a BF very close to 1, one does not have to update his or her priors. If one holds, for example, equal priors (p(H1) = p(H0) = .5), these probabilities do not change after having seen the data of the original study. With these data, it is not possible to decide between H0 and H1, and being so close to 1, this BF is not even “anectdotal evidence” for H1 (although the original study was just skirting the boundary of significance, p = .06).
For the replication data, the situation looks different. The Bayes factor here is 0.11. That means, H0 is (1 / 0.11) = 9 times more likely than H1. A of 0.11 would be labelled “moderate to strong evidence” for H0. If you had equals priors before, you should update your belief for H1 to 10% and for H0 to 90% (Berger, 2006).
To summarize, neither the original nor the replication study show evidence for H1. In contrast, the replication study even shows quite strong evidence for H0.
A more detailed look at the data using robust statistics
Parametric tests, like the ANOVA employed in the original and the replication study, rest on assumptions. Unfortunately, these assumptions are very rarely met (Micceri, 1989), and ANOVA etc. are not as robust against these violations as many textbooks suggest (Erceg-Hurn & Mirosevic, 2008). Fortunately, over the last 30 years robust statistical methods have been developed that do not rest on such strict assumptions.
In the presence of violations and outliers, these robust methods have much lower Type I error rates and/or higher power than classical tests. Furthermore, a key advantage of these methods is that they are designed in a way that they are equally efficient compared to classical methods, even if the assumptions are not violated. In a nutshell, when using robust methods, there nothing to lose, but a lot to gain.
Comparing the central tendency of two groups
A robust alternative to the independent group t test would be to compare the trimmed means via percentile bootstrap. This method is robust against outliers and does not rest on parametric assumptions. Here we find a p value of .106 for the original study and p = .94 for the replication study. Hence, the same picture: No evidence against the H0.
Comparing other-than-central tendencies between two groups, aka.: Comparing extreme quantiles between groups
When comparing data from two groups, approximately 99.6% of all psychological research compares the central tendency (that is a subjective estimate).
In some cases, however, it would be sensible to compare other parts of the distributions.For example, an intervention could have effects only on slow reaction times (RT), but not on fast or medium RTs. Similarly, priming could have an effect only on very high responses, but not on low and average responses. Measures of central tendency might obscure or miss this pattern.
And indeed, descriptively there (only) seems to be a priming effect in the “extremely wrong pole” (large numbers on the x axis) of the original study (i.e., the black density line is higher than the red on at “7” and “8” ratings):
This visual difference can be tested. Here, I employed the
qcomhd function from the WRS package (Wilcox, Erceg-Hurn, Clark, & Carlson, 2013). This method looks whether two samples differ in several quantiles (not only the central tendency). For an introduction to this method, see this blog post.
Here’s the result when comparing the 10th, 30th, 50th, 70th, and 90th quantile:
1 0.1 20 20 3.86 3.15 0.712 -1.077 2.41 0.0500 0.457 NO
2 0.3 20 20 4.92 4.51 0.410 -0.341 1.39 0.0250 0.265 NO
3 0.5 20 20 5.76 5.03 0.721 -0.285 1.87 0.0167 0.197 NO
4 0.7 20 20 6.86 5.70 1.167 0.023 2.05 0.0125 0.047 NO
5 0.9 20 20 7.65 6.49 1.163 0.368 1.80 0.0100 0.008 YES
(Please note: Estimating 5 quantiles from 20 data points is not quite the epitome of precision. So treat this analysis with caution.)
As multiple comparison are made, the Benjamini-Hochberg-correction for the p value is applied. This correction gives new critical p values (column
p_crit) to which the actual p values (column
p.value) have to be compared. One quantile survives the correction: the 90th quantile. That means that there are fewer extreme answers in the cleanliness priming group than in the control group
This finding, of course, is purely exploratory, and as any other exploratory finding it awaits cross-validation in a fresh data set. Luckily, we have the replication data set! Let’s see whether we can replicate this effect.
The answer is: no. Not even a tendency:
1 0.1 102 106 4.75 4.88 -0.1309 -0.705 0.492 0.0125 0.676 NO
2 0.3 102 106 6.00 6.12 -0.1152 -0.571 0.386 0.0250 0.699 NO
3 0.5 102 106 6.67 6.61 0.0577 -0.267 0.349 0.0500 0.737 NO
4 0.7 102 106 7.11 7.05 0.0565 -0.213 0.411 0.0167 0.699 NO
5 0.9 102 106 7.84 7.73 0.1111 -0.246 0.431 0.0100 0.549 NO
Here’s a plot of the results:
From the Bayes factor analysis, both the original and the replication study do not show evidence for the H1. The replication study actually shows moderate to strong evidence for the H0.
If anything, the original study shows some exploratory evidence that only the high end of the answer distribution (around the 90th quantile) is reduced by the cleanliness priming – not the central tendency. If one wants to interpret this effect, it would translate to: “Cleanliness primes reduce extreme morality judgements (but not average or low judgements)”. This exploratory effect, however, could not be cross-validated in the better powered replication study.
Recently, Silberzahn, Uhlmann, Martin, and Nosek proposed “crowdstorming a data set”, in cases where a complex data set calls for different analytical approaches. Now, a simple two group experimental design, usually analyzed with a t test, doesn’t seem to have too much complexity – still it is interesting how different analytical approaches highlight different aspects of the data set.
And it is also interesting to see that the majority of diverse approaches comes to the same conclusion: From this data base, we can conclude that we cannot conclude that the H0 is wrong (This sentence, a hommage to Cohen, 1990, was for my Frequentist friends ;-)).
And, thanks to Bayesian approaches, we can say (and even understand): There is strong evidence that the H0 is true. Very likely, there is no priming effect in this paradigm.
PS: Celebrate open science. Without open data, all of this would not be possible.
## This is a reanalysis of raw data from
## - Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience cleanliness reduces the severity of moral judgments. Psychological Science, 19(12), 1219-1222.
## - Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does Cleanliness Influence Moral Judgments? A Direct Replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45, 209-215.
## Read raw data, provided on Open Science Framework
# - https://osf.io/4cs3k/
# - https://osf.io/yubaf/
dat1 <- read.spss("raw/Schnall_Benton__Harvey_2008_Study_1_Priming.sav", to.data.frame=TRUE)
dat2 <- read.spss("raw/Exp1_Data.sav", to.data.frame=TRUE)
dat2 <- dat2[dat2[, "filter_."] == "Selected", ]
dat2$condition <- factor(dat2$Condition, labels=c("neutral priming", "clean priming"))
# build composite scores from the 6 vignettes:
dat1$DV <- rowMeans(dat1[, c("dog", "trolley", "wallet", "plane", "resume", "kitten")])
dat2$DV <- rowMeans(dat2[, c("Dog", "Trolley", "Wallet", "Plane", "Resume", "Kitten")])
# define shortcuts for DV in each condition
neutral <- dat1$DV[dat1$condition == "neutral priming"]
clean <- dat1$DV[dat1$condition == "clean priming"]
neutral2 <- dat2$DV[dat2$condition == "neutral priming"]
clean2 <- dat2$DV[dat2$condition == "clean priming"]
## Original analyses with t-tests/ ANOVA
# Original study:
# Some descriptives ...
# Run the ANOVA from Schnall et al. (2008)
a1 <- aov(DV ~ condition, dat1)
summary(a1) # p = .0644
# --> everything as in original publication
# Replication study
a2 <- aov(DV ~ condition, dat2)
summary(a2) # p = .947
# --> everything as in replication report
## Bayes factor analyses
ttestBF(neutral, clean, rscale=1) # BF_10 = 1.08
ttestBF(neutral2, clean2, rscale=1) # BF_10 = 0.11
## Robust statistics
# robust tests: group difference of central tendency
# percentile bootstrap for comparing measures of location:
# 20% trimmed mean
trimpb2(neutral, clean) # p = 0.106 ; CI: [-0.17; +1.67]
trimpb2(neutral2, clean2) # p = 0.9375; CI: [-0.30; +0.33]
medpb2(neutral, clean) # p = 0.3265; CI: [-0.50; +2.08]
medpb2(neutral2, clean2) # p = 0.7355; CI: [-0.33; +0.33]
# Comparing other quantiles (not only the central tendency)
# plot of densities
plot(density(clean, from=1, to=8), ylim=c(0, 0.5), col="red", main="Original data", xlab="Composite rating")
lines(density(neutral, from=1, to=8), col="black")
legend("topleft", col=c("black", "red"), lty="solid", legend=c("neutral", "clean"))
plot(density(clean2, from=1, to=8), ylim=c(0, 0.5), col="red", main="Replication data", xlab="Composite rating")
lines(density(neutral2, from=1, to=8), col="black")
legend("topleft", col=c("black", "red"), lty="solid", legend=c("neutral", "clean"))
# Compare quantiles of original study ...
qcomhd(neutral, clean, q=seq(.1, .9, by=.2), xlab="Original: Neutral Priming", ylab="Neutral - Clean")
# Compare quantiles of replication study
qcomhd(neutral2, clean2, q=seq(.1, .9, by=.2), xlab="Replication: Neutral Priming", ylab="Neutral - Clean")
Berger, J. O. (2006). Bayes factors. In S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic, & N. L. Johnson (Eds.), Encyclopedia of Statistical Sciences, vol. 1 (2nd ed.) (pp. 378–386). Hoboken, NJ: Wiley.
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008). Modern robust statistical methods: An easy way to maximize the accuracy and power of your research. American Psychologist, 63, 591–601.
Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166. doi:10.1037/0033-2909.105.1.156
Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience cleanliness reduces the severity of moral judgments. Psychological Science, 19(12), 1219–1222.
Wilcox, R. R., Erceg-Hurn, D. M., Clark, F., & Carlson, M. (2013). Comparing two independent groups via the lower and upper quantiles. Journal of Statistical Computation and Simulation, 1–9. doi:10.1080/00949655.2012.754026
Wilcox, R.R., & Schönbrodt, F.D. (2014). The WRS package for robust statistics in R (version 0.25.2). Retrieved from https://github.com/nicebread/WRS
These days psychology really is exciting, and I do not mean the Förster case …
In May 2014 a special issue full of replication attempts has been released – all open access, all raw data released! This is great work, powered by the open science framework and from my point of view a major leap forward.
One of the replication attempts, about wether “Cleanliness Influence Moral Judgments” generated a lot of heat in the social media. This culminated in a blog post by one of the replicators, Brent Donnellan, an independent analysis of the data by Chris Fraley, and finally a long post by Simone Schnall who is the original first author of the effect, which generated a lot of comments.
Here’s my personal summary, conclusions, and insights I gained from the debate (Much of this has been stated by several other commenters, so this is more the wisdom of the crowd than my own insights).
1. Accept uncertainty in scientific findings.
As long as we stick to the p < .05 ritual, 1 in 20 studies will produce false positive results if there is no effect in the population. Depending on the specific statistical test, the degree of violation of the assumptions of this test, and the amount of QRPs you apply, the actual Type I error rate can be both lower and higher than the nominal 5% (in practice, I’d bet on “higher”).
We know how to fix this – e.g., Bayesian statistics (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), “revised standards” (i.e., use p < .005 as magic threshold; Johnson, 2013), or focus on accuracy in parameter estimation instead of NHST (Maxwell, Kelley, & Rausch, 2008; Schönbrodt & Perugini, 2013).
A single study is hardly ever conclusive. Hence, a way to deal with uncertainty is to meta-analyze and to combine evidence. Let’s make a collaborative effort to increase knowledge, not (only) personal fame.
2. A failed replication does not imply fraud, QRP, or lack of competence of the original author.
This has been emphasized several times, and, as far as I read the papers of the special issue, no replicator made this implication. In contrast, the discussions were generally worded very cautious. Look in the original literature of the special issue, and you will hardly (or not at all) find evidence for “replication bullying”.
The nasty false-positive monster is lurking around for everybody of us. And when, someday, one of my studies cannot be replicated, then, of course I’ll be pissed off at first, but then …
It does *not* imply that I am a bad researcher.
It does *not* imply that the replicator is a “bully”.
But it means that we increased knowledge a little bit.
3. Transparency fosters progress.
4. Do I have to ask Galileo before I throw a stone from a tower? No.
Or, as Dale Barr (@dalejbarr) tweeted: “If you publish a study demonstrating an effect, you don’t OWN that effect.”
As several others argued, it usually is very helpful, sensible, and fruitful to include the original authors in the replication effort. In an ideal world we make a collaborative effort to increase knowledge, and a very good example for a “best practice adversarial collaboration”, see the paper and procedure by Dora Matzke and colleagues (2013). Here, proponents and skeptics of an effect worked, together with an impartial referee, on a confirmatory study.
To summarize, involvement of original authors is nice and often fruitful, but definitely not necessary.
5. Celebrate the debate.
Remember the old times where “debate” was carried out with a time-lag of at least 6 months (until your comment was printed in the journal, if you were lucky), and usually didn’t happen at all?
I *love* reading and participating the active debate. This feels so much more like science than what we used to do. What happens here is not a shitstorm – it’s a debate! And the majority of comments really is directed at the issue and has a calm and constructive tone.
So, use the new media for post-publication-review, #HIBAR (Had I Been A Reviewer), and real exchange. The rules of the game seem to change, slowly.