*This is a post-publication peer review (HIBAR: “Had I Been A Reviewer”) of the following paper:*

*Levi, M., Li, K., & Zhang, F. (2010). Deal or no deal: Hormones and the mergers and acquisitions game. Management Science 56, 1462 -1483.*

*A citeable version of this post-publication peer review can be found at SSRN: *

*Schönbrodt, Felix D., Why Using Age as a Proxy for Testosterone is a Bad Deal (March 26, 2012). Available at SSRN: http://ssrn.com/abstract=2028882 or http://dx.doi.org/10.2139/ssrn.2028882*

In their article “Deal or No Deal: Hormones and the Mergers and Acquisitions Game,” Levi, Li, and Zhang (2010) claimed that they investigated the effect of testosterone on CEOs’ decisions in mergers and acquisitions. However, they did not measure testosterone levels directly. Rather, they tried to use CEO age as a proxy, based on a previously documented negative correlation between age and testosterone level. In this comment, I argue that it is not reasonable to use age as a proxy for testosterone, and that Levi et al.’s study does not tell us anything about testosterone. General remarks on using proxy variables are given.

In their article “Deal or No Deal: Hormones and the Mergers and Acquisitions Game,” Levi, Li, and Zhang (2010) investigated the research question of whether the hormone testosterone (T) has an impact on decisions in mergers and acquisitions (M&As). Based on experimental results that T has an effect on behavior in ultimatum games (Burnham, 2007), Levi et al. hypothesized that CEOs with higher T levels should show more aggressive/dominant behavior in M&As. To investigate this hypothesis, the authors assembled a data set with 357 M&As and several economic variables related to them (e.g., the size of the target firm, the board size, and several other economic indicators). As they could not assess the T levels of the CEOs directly, they “[...] have suggested an alternative: specifically, to proxy testosterone by age” (p. 1476). Therefore, as the authors admitted themselves, their reasoning was based on a central assumption: “The validity of this approach clearly depends on the extent of the association of hormone levels with age.” (p. 1476). To summarize their findings, a significant but small effect of age on M&A decisions was found: younger CEOs made more bid withdrawals and more tender offers than their older counterparts (which has been interpreted as more dominant behavior). As the story has received wide press coverage, for example, in the Wall Street Journal, Financial Times, Time Magazine, and the Los Angeles Times, I feel the need to make some clarifications.

Empirically, Levi et al. have shown an effect of CEO age on the outcome of M&As. From the title to the conclusion of their article, however, they refer to the effect of testosterone (e.g., “[...] in M&As the testosterone of both parties could influence the course and outcome of negotiations,” p. 1463; “[...] we consider whether testosterone influences the likelihood that offers made are subsequently withdrawn,” p.1466; “This finding strongly supports an association between testosterone, as proxied by the bidder male CEO age, and M&As,” p. 1469).

In the following commentary, I argue that it is not appropriate to use age as a proxy for T level and that the conclusions of Levi et al. are taking it way too far. For the clarity of my arguments, I will focus only on the strongest reported effect. For all weaker effects, the same reasoning applies even more.

Is it a reasonable hypothesis to expect more dominant M&A behavior from CEOs with higher T levels? Early investigations with animals have shown a relation between testosterone level and aggressive or dominant behavior (e.g., Wingfield, Hegner, Dufty, & Ball, 1990). Recent review articles and meta-analyses on human testosterone, however, have shown that the effects in humans are rather small. For example, in a meta-analysis on the relation of male human aggression and T level (N = 9760; Archer, Graham-Kevan, & Davies, 2005; Book, Starzyk, & Quinsey, 2001), the weighted correlation was only .08. Other meta-analyses on the relation of T to dominance (weighted = .13) and to a challenge in a sports competition (weighted = .18) have supported this finding of a small effect (for an overview, see Archer, 2006). To summarize, if M&As are seen as competitive situations, indeed, an effect of T could be expected – but the relation is probably much less pronounced than common sense might suggest.

The central assumption in their article is that T level can be proxied by age. How strong is that relation? A meta-analysis of 23 studies reporting the correlation between age and T level (N = 1900) revealed that the average correlation between these variables was -.18 (Gray, Berlin, McKinlay, & Longcope, 1991). That means only 3% of the variation in T level can be explained by age. Levi et al. refer to another study that shows a remarkably stronger correlation of -.50 (Harman, Metter, Tobin, Pearson, & Blackman, 2001).

All articles cited by Levi et al. report the correlation for a broad age range (e.g., 24 – 90 years, Ferrini & Barrett-Conner, 1998; 23 – 91 years, Harman et al., 2001). Given that 90% of CEOs in Levi et al.’s sample were between 46 and 64, a massive range restriction is present, which presumably lowers the correlation in that age range even more. Indeed, the scatter plots in Harman et al. (2001) strongly suggest that the correlation is mainly driven by the very young and very old participants.

But regardless of whether the true correlation is closer to -.18 or to -.50: Are these correlations high enough that age is warranted as a valid proxy for T level? As we will see in the next section, the answer is no.

In their conclusions, the authors argue as though they had established a causal effect of T on M&A decisions. However, they had not even established a correlation between T and M&A behavior. Their approach seems to be something like a “triangulation” of an unobserved correlation by the knowledge of two other correlations. Indeed, there are dependencies and constraints in the relation of the bivariate correlations between three variables. If two of the three correlations are given ( and ), the possible range of the third correlation, , is constrained by following equation (Olkin, 1981):

(1)

Figure 1 graphically represents the relationship of variables given by this equation. The left plot shows the upper boundary of , the right plot the lower boundary of . As one can see, rather high values for either , , or both, have to be present to imply a positive sign (i.e., a lower boundary > 0) of .

Given the observed estimates of = -.50 and = -.12, the range of possible values for goes from -.80 to +.92. (The position is marked by the asteriks in Figure 1.) If the probably more realistic estimate from the meta-analysis is used ( = -.18; Gray et al., 1991), the possible range for goes from -.95 to +1.00. These calculations clearly show that there is no argument to expect a positive correlation between T and M&A decisions. Actually, with the given data set, no conclusions about T can be drawn at all.

Given these analyses, it should be clear that one cannot speak of a T effect based on these data. What about the age effect reported in this article? The authors wrote that, “[... m]otivated by the studies of population testosterone levels reviewed earlier, we use the age of 45 years as the cut-off to separate young male CEOs from the rest.” (p. 1467). This seems to be a problematic choice to me. First of all, none of the cited studies gives a hint about why the age of 45 should be a particularly meaningful cut point. None of the reviewed studies suggests a significant break or stronger decline of T at that age. Furthermore, that cut point leads to a very skewed distribution of 16 “young” vs. 341 “old” CEOs.

The strongest reported effect was that of the dichotomized variable “CEO is young” (i.e., younger than 45) on bid withdrawals. Based on the reported descriptive statistics and correlations, it can be computed that 5 young CEOs withdrew, whereas 11 did not. Concerning old CEOs, 40 out of 341 withdrew. This distribution led to the reported Pearson correlation of = .12 ( = .02). If only one young CEO would not have withdrawn, the correlation would have been nonsignificant ( = .08, = .13). If only three young CEOs would not have withdrawn, age and withdrawing behavior would be completely unrelated. Hence, as Levi et al.’s conclusions stand and fall on the decision of one single CEO, these results do not seem very robust to me. Why is age categorized at all? Methodological papers on the topic clearly suggest not to dichotomize if a predictor variable can be used on an interval or ratio scale (e.g., Royston, Altman, & Sauerbrei, 2006).

Referring to several meta-analyses, it could be shown that the relation between T and dominant human behavior is much less pronounced than suggested by Levi and colleagues. Likewise, the relation between age and testosterone, especially in the restricted age range of their sample, is presumably close to zero. From my reading of the empirical evidence, the article should be rewritten as “Age and the mergers and acquisitions game,” and it should be acknowledged that the age effect, although significant, has a small effect size.

Of course, age is not a psychological variable per se – it is always a proxy for a true causal factor: “And although it is unquestionably useful to find that a phenomenon covaries with age, neither age nor the related variable time is a causal variable; changes occur in time, but not as a result of time.” (Hartmann & George, 1999, p. 132). Hence, it would be feasible to spend one or two paragraphs in the discussion speculating about the potential causal role of testosterone (and several other age-related variables). In that article, however, from the main title to the conclusion, the authors argue that T is the causal factor. Although the authors try to rule out some alternative age-related factors, the main criticism remains: With the current data, there is no way to provide evidence for or against a T hypothesis. It’s simply the wrong story for the data.

Based on the same data set, several other articles could have been written in the same style, for example, “Bicep strength and the M&A game.” Why bicep strength? The theory of embodied cognition predicts that physical dominance is a predictor for psychological dominance in negotiations (Barsalou, 2008). As CEO muscle strength cannot be measured directly, one could use CEO age as a proxy for muscle strength ( =.55; Lindle et al., 1997). Physically stronger CEOs, as proxied by age, would therefore be expected to make more dominant choices in M&As. This ad hoc hypothesis has the same argumentative structure and the same empirical justification as the “testosterone story,” and the creative reader can think of a multitude of other age-related variables that might have an influence on negotiation decisions (e.g., generational cohort effects in moral values, fluid intelligence, different educational levels between age groups, time spent in marriage, or the grayishness of the CEO’s hair).

To summarize, Levi et al. conclude that they “[...] have been able to conduct that age appears to be proxying for testosterone rather than experience, horizon, or some other effect” (p. 1478) and that their finding “strongly supports an association between testosterone, as proxied by the bidder male CEO age, and M&As.” (p. 1469). In the light of the analyses given above, that conclusion is not justified.

Archer, J. 2006. Testosterone and human aggression: An evaluation of the challenge hypothesis. Neuroscience & Biobehavioral Reviews 30(3) 319-345.

Archer, J., N. Graham-Kevan, M. Davies. 2005. Testosterone and aggression: A reanalysis of Book, Starzyk, and Quinsey’s (2001) study. Aggression and Violent Behavior 10(2) 241-261.

Barsalou, L.W. 2008. Grounded cognition. Annu. Rev. Psychol. 59 617–645.

Book, A.S., K.B. Starzyk, V.L. Quinsey. 2001. The relationship between testosterone and aggression: A meta-analysis. Aggression and Violent Behavior 6(6) 579–599.

Burnham, T. C. 2007. High-testosterone men reject low ultimatum game offers. Proceedings of the Royal Society B: Biological Sciences 274(1623) 2327-2330.

Cohen, J. 1992. A power primer. Psychological Bulletin 112(1) 155-159.

Ferrini, R. L., E. Barrett-Connor. 1998. Sex hormones and age: a cross-sectional study of testosterone and estradiol and their bioavailable fractions in community-dwelling men. American Journal of Epidemiology 147 750-754.

Gray, A., J. A. Berlin, J. B. McKinlay, C. Longcope. 1991. An examination of research design effects on the association of testosterone and male aging: results of a meta-analysis. Journal of Clinical Epidemiology 44(7) 671-684.

Harman, S. M., E. J. Metter, J. D. Tobin, J. Pearson, M. R. Blackman. 2001. Longitudinal effects of aging on serum total and tree testosterone levels in healthy men. Journal of Clinical Endocrinology & Metabolism 86(2) 724 -731.

Hartmann, D. P., T. P. George. 1999. Design, measurement, and analysis in developmental research. M. H. Bornstein, M. E. Lamb, ed. Developmental psychology: An advanced textbook (4th ed.). Mahwah, NJ US: Lawrence Erlbaum Associates Publishers, 125-195.

Levi, M., K. Li, F. Zhang. 2010. Deal or no deal: Hormones and the mergers and acquisitions game. Management Science 56(9) 1462 -1483.

Lindle, R. S., E. J. Metter, N. A. Lynch, J. L. Fleg, J. L. Fozard, J. Tobin, T. A. Roy, B. F. Hurley. 1997. Age and gender comparisons of muscle strength in 654 women and men aged 20–93 yr. Journal of Applied Physiology 83(5) 1581-1587.

Olkin, I. 1981. Range restrictions for product-moment correlation matrices. Psychometrika 46(4) 469-472.

Royston, P., D. G. Altman, W. Sauerbrei. 2006. Dichotomizing continuous predictors in multiple regression: A bad idea. Statistics in Medicine 25(1) 127-141.

Wingfield, J. C., R. E. Hegner, A. M. Dufty, G. F. Ball. 1990. The ‘Challenge Hypothesis’: Theoretical Implications for Patterns of Testosterone Secretion, Mating Systems, and Breeding Strategies. The American Naturalist 136(6) 829-846.

]]>

I want to present a re-analysis of the raw data from two studies that investigated whether physical cleanliness reduces the severity of moral judgments – from the original study (*n* = 40; Schnall, Benton, & Harvey, 2008), and from a direct replication (*n* = 208, Johnson, Cheung, & Donnellan, 2014). Both data sets are provided on the Open Science Framework. All of my analyses are based on the composite measure as dependent variable.

This re-analsis follows previous analyses by Tal Yarkoni, Yoel Inbar, and R. Chris Fraley, and is focused on one question: *What can we learn from the data when we apply modern (i.e., Bayesian and robust) statistical approaches?*

The complete and reproducible R code for these analyses is at the end of the post.

Disclaimer 1: This analysis assumes that the studies from which data came from were internally valid. Of course the garbage-in-garbage-out principle holds. But as the original author reviewed the experimental material of the replication study and gave her OK, I assume that the relication data is as valid as the original data.

Disclaimer 2: I am not going to talk about tone, civility, or bullying here. Although these are important issues, a lot has been already written about it, including apologies from one side of the debate (not from the other, yet), etc. For nice overviews over the debate, see for example a blog post by Tal Yarkoni). I am completely unemotional about these data. False positives do exists, I am sure I had my share of them, and replication is a key element of science. I do not suspect anybody of anything – I just look at the data.

, and the summary provided byThat being said, let’s get to business:

The BF is a continuous measure of evidence for H0 or for H1, and quantifies, “how much more likely is H1 compared to H0″. Typically, a BF of at least 3 is requested to speak of evidence (i.e., the H1 should be at least 3 times more likely than the H0 to speak of evidence for an effect). For an introduction to Bayes factors see here, here, or here.

Using the BayesFactor package, it is simple to compute a Bayes factor (BF) for the group comparison. In the original study, the Bayes factor against the H0, , is 1.08. That means, data are 1.08 times more likely under the H1 (“there is an effect”) than under the H0 (“There is no effect”). As the BF is virtually 1, data occurred equally likely under both hypotheses.

What researchers actually are interested in is not p(Data | Hypothesis), but rather p(Hypothesis | Data). Using Bayes’ theorem, the former can be transformed into the latter by assuming prior probabilities for the hypotheses. The BF then tells one how to update one’s prior probabilities after having seen the data. Given a BF very close to 1, one does not have to update his or her priors. If one holds, for example, equal priors (p(H1) = p(H0) = .5), these probabilities do not change after having seen the data of the original study. With these data, it is not possible to decide between H0 and H1, and being so close to 1, *this BF is not even “anectdotal evidence” for H1 *(although the original study was just skirting the boundary of significance, *p* = .06).

For the replication data, the situation looks different. The Bayes factor here is 0.11. That means, H0 is (1 / 0.11) = 9 times more likely than H1. A of 0.11 would be labelled “moderate to strong evidence” for H0. If you had equals priors before, you should update your belief for H1 to 10% and for H0 to 90% (Berger, 2006).

*To summarize, neither the original nor the replication study show evidence for H1. In contrast, the replication study even shows quite strong evidence for H0.*

Parametric tests, like the ANOVA employed in the original and the replication study, rest on assumptions. Unfortunately, these assumptions are very rarely met (Micceri, 1989), and ANOVA etc. are not as robust against these violations as many textbooks suggest (Erceg-Hurn & Mirosevic, 2008). Fortunately, over the last 30 years robust statistical methods have been developed that do not rest on such strict assumptions.

In the presence of violations and outliers, these robust methods have much lower Type I error rates and/or higher power than classical tests. Furthermore, a key advantage of these methods is that they are designed in a way that they are equally efficient compared to classical methods, even if the assumptions are *not* violated. In a nutshell, when using robust methods, there nothing to lose, but a lot to gain.

Rand Wilcox pioneered in developing many robust methods (see, for example, this blog post by him), and the methods are implemented in the WRS package for R (Wilcox & Schönbrodt, 2014).

A robust alternative to the independent group *t* test would be to compare the trimmed means via percentile bootstrap. This method is robust against outliers and does not rest on parametric assumptions. Here we find a *p* value of .106 for the original study and *p* = .94 for the replication study. Hence, the same picture: No evidence against the H0.

When comparing data from two groups, approximately 99.6% of all psychological research compares the *central tendency* (that is a subjective estimate).

In some cases, however, it would be sensible to compare other parts of the distributions.For example, an intervention could have effects only on slow reaction times (RT), but not on fast or medium RTs. Similarly, priming could have an effect only on very high responses, but not on low and average responses. Measures of central tendency might obscure or miss this pattern.

And indeed, descriptively there (only) seems to be a priming effect in the “extremely wrong pole” (large numbers on the x axis) of the original study (i.e., the black density line is higher than the red on at “7” and “8” ratings):

This visual difference can be tested. Here, I employed the [cci]qcomhd[/cci] function from the WRS package (Wilcox, Erceg-Hurn, Clark, & Carlson, 2013). This method looks whether two samples differ in several quantiles (not only the central tendency). For an introduction to this method, see this blog post.

Here’s the result when comparing the 10th, 30th, 50th, 70th, and 90th quantile:

[cc lang="rsplus" escaped="true"]

q n1 n2 est.1 est.2 est.1_minus_est.2 ci.low ci.up p_crit p.value signif

1 0.1 20 20 3.86 3.15 0.712 -1.077 2.41 0.0500 0.457 NO

2 0.3 20 20 4.92 4.51 0.410 -0.341 1.39 0.0250 0.265 NO

3 0.5 20 20 5.76 5.03 0.721 -0.285 1.87 0.0167 0.197 NO

4 0.7 20 20 6.86 5.70 1.167 0.023 2.05 0.0125 0.047 NO

5 0.9 20 20 7.65 6.49 1.163 0.368 1.80 0.0100 0.008 YES

[/cc]

(Please note: Estimating 5 quantiles from 20 data points is not quite the epitome of precision. So treat this analysis with caution.)

As multiple comparison are made, the Benjamini-Hochberg-correction for the *p* value is applied. This correction gives new critical *p* values (column [cci]p_crit[/cci]) to which the actual *p* values (column [cci]p.value[/cci]) have to be compared. One quantile survives the correction: the 90th quantile. That means that there are fewer extreme answers in the cleanliness priming group than in the control group

This finding, of course, is purely exploratory, and as any other exploratory finding it awaits cross-validation in a fresh data set. Luckily, we have the replication data set! Let’s see whether we can replicate this effect.

The answer is: no. Not even a tendency:

[cc lang="rsplus" escaped="true"]

q n1 n2 est.1 est.2 est.1_minus_est.2 ci.low ci.up p_crit p.value signif

1 0.1 102 106 4.75 4.88 -0.1309 -0.705 0.492 0.0125 0.676 NO

2 0.3 102 106 6.00 6.12 -0.1152 -0.571 0.386 0.0250 0.699 NO

3 0.5 102 106 6.67 6.61 0.0577 -0.267 0.349 0.0500 0.737 NO

4 0.7 102 106 7.11 7.05 0.0565 -0.213 0.411 0.0167 0.699 NO

5 0.9 102 106 7.84 7.73 0.1111 -0.246 0.431 0.0100 0.549 NO

[/cc]

Here’s a plot of the results:

From the Bayes factor analysis, both the original and the replication study do not show evidence for the H1. The replication study actually shows moderate to strong evidence for the H0.

If anything, the original study shows some exploratory evidence that only the high end of the answer distribution (around the 90th quantile) is reduced by the cleanliness priming – not the central tendency. If one wants to interpret this effect, it would translate to: “Cleanliness primes reduce extreme morality judgements (but not average or low judgements)”. This exploratory effect, however, could not be cross-validated in the better powered replication study.

Recently, Silberzahn, Uhlmann, Martin, and Nosek proposed “crowdstorming a data set”, in cases where a complex data set calls for different analytical approaches. Now, a simple two group experimental design, usually analyzed with a *t* test, doesn’t seem to have too much complexity – still it is interesting how different analytical approaches highlight different aspects of the data set.

And it is also interesting to see that the majority of diverse approaches comes to the same conclusion: From this data base, we can conclude that we cannot conclude that the H0 is wrong (This sentence, a hommage to Cohen, 1990, was for my Frequentist friends ;-)).

And, thanks to Bayesian approaches, we can say (and even understand): *There is strong evidence that the H0 is true. Very likely, there is no priming effect in this paradigm.*

PS: Celebrate open science. Without open data, all of this would not be possible.

[cc lang="rsplus" escaped="true" collapse="true"]

## (c) 2014 Felix Schönbrodt

## http://www.nicebread.de

##

## This is a reanalysis of raw data from

## – Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience cleanliness reduces the severity of moral judgments. Psychological Science, 19(12), 1219-1222.

## – Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does Cleanliness Influence Moral Judgments? A Direct Replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45, 209-215.

## ======================================================================

## Read raw data, provided on Open Science Framework

# – https://osf.io/4cs3k/

# – https://osf.io/yubaf/

## ======================================================================

library(foreign)

dat1 <- read.spss("raw/Schnall_Benton__Harvey_2008_Study_1_Priming.sav", to.data.frame=TRUE)

dat2 <- read.spss("raw/Exp1_Data.sav", to.data.frame=TRUE)

dat2 <- dat2[dat2[, "filter_."] == "Selected", ]

dat2$condition <- factor(dat2$Condition, labels=c("neutral priming", "clean priming"))

table(dat2$condition)

# build composite scores from the 6 vignettes:

dat1$DV <- rowMeans(dat1[, c("dog", "trolley", "wallet", "plane", "resume", "kitten")])

dat2$DV <- rowMeans(dat2[, c("Dog", "Trolley", "Wallet", "Plane", "Resume", "Kitten")])

# define shortcuts for DV in each condition

neutral <- dat1$DV[dat1$condition == "neutral priming"]

clean <- dat1$DV[dat1$condition == "clean priming"]

neutral2 <- dat2$DV[dat2$condition == "neutral priming"]

clean2 <- dat2$DV[dat2$condition == "clean priming"]

## ======================================================================

## Original analyses with t-tests/ ANOVA

## ======================================================================

# ---------------------------------------------------------------------

# Original study:

# Some descriptives ...

mean(neutral)

sd(neutral)

mean(clean)

sd(clean)

# Run the ANOVA from Schnall et al. (2008)

a1 <- aov(DV ~ condition, dat1)

summary(a1) # p = .0644

# --> everything as in original publication

# ———————————————————————

# Replication study

mean(neutral2)

sd(neutral2)

mean(clean2)

sd(clean2)

a2 <- aov(DV ~ condition, dat2)

summary(a2) # p = .947

# --> everything as in replication report

## ======================================================================

## Bayes factor analyses

## ======================================================================

library(BayesFactor)

ttestBF(neutral, clean, rscale=1) # BF_10 = 1.08

ttestBF(neutral2, clean2, rscale=1) # BF_10 = 0.11

## ======================================================================

## Robust statistics

## ======================================================================

library(WRS)

# ———————————————————————

# robust tests: group difference of central tendency

# percentile bootstrap for comparing measures of location:

# 20% trimmed mean

trimpb2(neutral, clean) # p = 0.106 ; CI: [-0.17; +1.67]

trimpb2(neutral2, clean2) # p = 0.9375; CI: [-0.30; +0.33]

# median

medpb2(neutral, clean) # p = 0.3265; CI: [-0.50; +2.08]

medpb2(neutral2, clean2) # p = 0.7355; CI: [-0.33; +0.33]

# ———————————————————————

# Comparing other quantiles (not only the central tendency)

# plot of densities

par(mfcol=c(1, 2))

plot(density(clean, from=1, to=8), ylim=c(0, 0.5), col=”red”, main=”Original data”, xlab=”Composite rating”)

lines(density(neutral, from=1, to=8), col=”black”)

legend(“topleft”, col=c(“black”, “red”), lty=”solid”, legend=c(“neutral”, “clean”))

plot(density(clean2, from=1, to=8), ylim=c(0, 0.5), col=”red”, main=”Replication data”, xlab=”Composite rating”)

lines(density(neutral2, from=1, to=8), col=”black”)

legend(“topleft”, col=c(“black”, “red”), lty=”solid”, legend=c(“neutral”, “clean”))

# Compare quantiles of original study …

par(mfcol=c(1, 2))

qcomhd(neutral, clean, q=seq(.1, .9, by=.2), xlab=”Original: Neutral Priming”, ylab=”Neutral – Clean”)

abline(h=0, lty=”dotted”)

# Compare quantiles of replication study

qcomhd(neutral2, clean2, q=seq(.1, .9, by=.2), xlab=”Replication: Neutral Priming”, ylab=”Neutral – Clean”)

abline(h=0, lty=”dotted”)

[/cc]

*References*

Berger, J. O. (2006). Bayes factors. In S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic, & N. L. Johnson (Eds.), Encyclopedia of Statistical Sciences, vol. 1 (2nd ed.) (pp. 378–386). Hoboken, NJ: Wiley.

Erceg-Hurn, D. M., & Mirosevich, V. M. (2008). Modern robust statistical methods: An easy way to maximize the accuracy and power of your research. American Psychologist, 63, 591–601.

Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166. doi:10.1037/0033-2909.105.1.156

Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience cleanliness reduces the severity of moral judgments. Psychological Science, 19(12), 1219–1222.

Wilcox, R. R., Erceg-Hurn, D. M., Clark, F., & Carlson, M. (2013). Comparing two independent groups via the lower and upper quantiles. *Journal of Statistical Computation and Simulation*, 1–9. doi:10.1080/00949655.2012.754026

Wilcox, R.R., & Schönbrodt, F.D. (2014). The WRS package for robust statistics in R (version 0.25.2). Retrieved from https://github.com/nicebread/WRS

]]>Last week a special issue full of replication attempts has been released – all open access, all raw data released! This is great work, powered by the open science framework and from my point of view a major leap forward.

One of the replication attempts, about wether “Cleanliness Influence Moral Judgments” generated a lot of heat in the social media. This culminated in a blog post by one of the replicators, Brent Donnellan, an independent analysis of the data by Chris Fraley, and finally a long post by Simone Schnall who is the original first author of the effect, which generated a lot of comments.

Here’s my personal summary, conclusions, and insights I gained from the debate (Much of this has been stated by several other commenters, so this is more the wisdom of the crowd than my own insights).

As long as we stick to the p < .05 ritual, ~~at least 1 in 20 studies will produce false positive results. No need to evoke ceiling effects, QRP, or other bad things: 1 in 20 ~~*valid* studies which were *perfectly planned* from *very skilled* researchers will produce false positives!

*Update 2014-5-27: As @johnmyleswhite and @hardsci noticed on Twitter: This statement was wrong (or at least: only correct under specific conditions). They really got me. (reminder for myself: think twice before you ever write anything about p values again!). Here’s an updated version (maybe my friends at Twitter want to check):*

As long as we stick to the p < .05 ritual, 1 in 20 studies will produce false positive results if there is no effect in the population. Depending on the specific statistical test, the degree of violation of the assumptions of this test, and the amount of QRPs you apply, the actual Type I error rate can be both lower and higher than the nominal 5% (in practice, I’d bet on “higher”). If, in contrast, there is an effect in the population, there cannot be false positives, and low *p* values correctly indicate evidence against the H0. To summarize, my original statement only applies if you assume that in psychology we only investigate null effects. Although I’m quite sceptical about effect sizes in psychology, this might be too harsh ;-).

(Well, that sounds less catchy then my original wording. Damn.)

We know how to fix this – e.g., Bayesian statistics (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), “revised standards” (i.e., use p < .005 as magic threshold; Johnson, 2013), or focus on accuracy in parameter estimation instead of NHST (Maxwell, Kelley, & Rausch, 2008; Schönbrodt & Perugini, 2013).

A single study is hardly ever conclusive. Hence, a way to deal with uncertainty is to meta-analyze and to combine evidence. Let’s make a collaborative effort to increase knowledge, not (only) personal fame.

This has been emphasized several times, and, as far as I read the papers of the special issue, no replicator made this implication. In contrast, the discussions were generally worded very cautious. Look in the original literature of the special issue, and you will hardly (or not at all) find evidence for “replication bullying”.

The nasty false-positive monster is lurking around for everybody of us. And when, someday, one of my studies cannot be replicated, then, of course I’ll be pissed off at first, but then …

It does *not* imply that I am a bad researcher.

It does *not* imply that the replicator is a “bully”.

But it means that we increased knowledge a little bit.

Without open data in OSF, we couldn’t even discuss about ceiling effects!

Or, as Dale Barr (@dalejbarr) tweeted: “If you publish a study demonstrating an effect, you don’t OWN that effect.”

As several others argued, it usually is very helpful, sensible, and fruitful to include the original authors in the replication effort. In an ideal world we make a collaborative effort to increase knowledge, and a very good example for a “best practice adversarial collaboration”, see the paper and procedure by Dora Matzke and colleagues (2013). Here, proponents and skeptics of an effect worked, together with an impartial referee, on a confirmatory study.

To summarize, involvement of original authors is nice and often fruitful, but definitely not necessary.

Remember the old times where “debate” was carried out with a time-lag of at least 6 months (until your comment was printed in the journal, if you were lucky), and usually didn’t happen at all?

I *love* reading and participating the active debate. This feels so much more like science than what we used to do. What happens here is not a shitstorm – it’s a debate! And the majority of comments really is directed at the issue and has a calm and constructive tone.

So, use the new media for post-publication-review, #HIBAR (Had I Been A Reviewer), and real exchange. The rules of the game seem to change, slowly.

]]>[1], [2], oder [3].

Facebook-Gruppe: https://www.facebook.com/akademischeJuniorposition

Folgende Abbildung sagt eigentlich alles:

(R.Kreckel, Habilitation vs. Tenure Track, Forschung und Lehre 01/2012, p.12)

Wenn ihr das unterstützt, dann unterschreibt online (kann man auch anonym machen), und leitet die Petition an eure Netzwerke weiter.

Es kann jeder unterschreiben – also auch eure Eltern, Freunde, etc!

While I agree to the main point of the post, I’d like to discuss and extend some of the conclusions. As the DataColada blog has no comments section, I’ll comment in my own blog …

“Does it make sense to push for effect size reporting when we run small samples? I don’t see how.”

“Properly powered studies teach you almost nothing about effect size.”

It is true that the ES estimate with *n*=20 will be utterly imprecise, and reporting this ES estimate could misguide readers who give too much importance to the point estimate and do not take the huge CI into account (maybe, because it has not been reported).

**Still, and here’s my disagreement, I’d argue that also small- n studies should report the point estimate (along with the CI)**, as a meta-analysis of many imprecise small-

Here’s an example – we simulate a population with a true Cohen’s d of 0.8. Then we look at three scenarios: a) a single small-*n* study with 20 participants, b) 400 participants, and c) 20 * *n*=20 studies which are meta-analyzed.

[cc lang="rsplus" escaped="true"]

# simulate population with true ES d=0.8

set.seed(0xBEEF1)

library(compute.es)

library(metafor)

library(ggplot2)

X1 <- rnorm(1000000, mean=0, sd=1)

X2 <- rnorm(1000000, mean=0, sd=1) – 0.8

## Compute effect size for …

# … a single n=20 study

x1 <- sample(X1, 20)

x2 <- sample(X2, 20)

(t1 <- t.test(x1, x2))

(ES1 <- tes(t1$statistic, 20, 20, dig=3))

[/cc]

This single small-*n* study has g = 1.337 [ 0.64 , 2.034] (g is an unbiased estimate of d). Quite biased, but the true ES is in the CI.

[cc lang="rsplus" escaped="true"]

# .. a single n=400 study

x1 <- sample(X1, 400)

x2 <- sample(X2, 400)

(t2 <- t.test(x1, x2))

(ES2 <- tes(t2$statistic, 400, 400, dig=3))

[/cc]

This single large-*n* study has g = 0.76 [ 0.616 , 0.903] . It close to the true ES, and has quite narrow CI (i.e., high precision).

Now, here’s a meta-analyses of 20*n=20 studies:

[cc lang="rsplus" escaped="true"]

# … 20x n=20 studies

dat <- data.frame()

for (i in 1:20) {

x1 <- sample(X1, 20)

x2 <- sample(X2, 20)

dat <- rbind(dat, data.frame(

study=i,

m1i=mean(x1),

m2i=mean(x2),

sd1i=sd(x1),

sd2i=sd(x2),

n1i=20, n2i=20

))

}

}

# Do a fixed-effect model meta-analysis

es <- escalc(“SMD”, m1i=m1i, m2i=m2i, sd1i=sd1i, sd2i=sd2i, n1i=n1i, n2i=n2i, data=dat, append=TRUE)

(meta <- rma(yi, vi, data=es, method=”FE”))

[/cc]

The meta-analysis reveals g = 0.775 [0.630; 0.920]. This has nearly exactly the same CI width as the n=400 study, and a slightly different ES estimate.

Here’s a plot of the results:

[cc lang="rsplus" escaped="true"]

# Plot results

res <- data.frame(

n = factor(c(“a) n=20″, “b) n=400″, “c) 20 * n=20\n(meta-analysis)”), ordered=TRUE),

point_estimate = c(ES1$d, ES2$d, meta$b),

ci.lower = c(ES1$l.d, ES2$l.d, meta$ci.lb),

ci.upper = c(ES1$u.d, ES2$u.d, meta$ci.ub)

)

ggplot(res, aes(x=n, y=point_estimate, ymin=ci.lower, ymax=ci.upper)) + geom_pointrange() + theme_bw() + xlab(“”) + ylab(“Cohen’s d”) + geom_hline(yintercept=0.8, linetype=”dotted”, color=”darkgreen”)[/cc]

To summarize, a single small-*n* study hardly teaches something about effect sizes – but many small-*n*‘s do. But meta-analyses are only possible, if the ES is reported.

“But just how big an

ndo we need to study effect size? I am about to show that the answer has four-digits.”

In Uri’s post (and the linked R code) the precision issue is approached from the power side – if you increase power, you also increase precision. But you can also directly compute the necessary sample size for a desired precision. This is called the AIPE-framework (“accuracy in parameter estimation”) made popular by Ken Kelley, Scott Maxwell, and Joseph Rausch (Kelley & Rausch, 2006; Kelley & Maxwell, 2003; Maxwell, Kelley, & Rausch, 2008). The necessary functions are implemented in the MBESS package for R. If you want a CI width of .10 around an expected ES of 0.5, you need 3170 participants:

[cc lang="rsplus" escaped="true"]

library(MBESS)

ss.aipe.smd(delta=0.5, conf.level=.95, width=0.10)

[1] 3170

[/cc]

The same point has been made from a Bayesian point of view in a blog post from John Kruschke: notice the sample size on the x-axis.

In our own analysis on how correlations evolve with increasing sample size (Schönbrodt & Perugini, 2013; see also blog post), we conclude that for typical effect sizes in psychology, you need 250 participants to get sufficiently accurate and stable estimates of the ES:

It’s certainly hard to give general guidelines how much precision is sensible, but here are our thoughts we based our stability analyses on. We used a CI-like “corridor of stability” (see Figure) with a half-width of w= .10, w=.15, and w=.20 (everything in the “correlation-metric”).

w = .20 was chosen for following reason: The average reported effect size in psychology is around *r* = .21 (Richard, Bond, & Stokes-Zoota, 2003). For this effect size, an accuracy of w = .20 would result in a CI that is “just significant” and does not include a reversal of the sign of the effect. Hence, with the typical effect sizes we are dealing with in psychology, a CI with a half-width > .20 would not make much sense.

w = .10 was chosen as it corresponds to a “small effect size” à la Cohen. This is arbitrary, of course, but at least some anchor. And w = .15 is just in between.

Using these numbers and an ES estimate of, say, *r* = .29, a just tolerable precision would be [.10; .46] (w = .20), a tolerable precision [.15; .42] (w = .15) and a moderate precision [.20; .38] (w = .10).

If we use this lower threshold of “just tolerable precision”, we would need in the two-sample group difference around 200-250 participants per group. While I am not sure whether we really need four-digit samples for typical scenarios, **I am sure that we need at least three-digit samples when we want to talk about “precision”.**

Regardless of the specific level of precision and method used, however, one thing is clear: **Accuracy does not come in cheaply**. We need much less participants for an hypothesis test (Is there a non-zero effect or not?) compared to an accurate estimate.

With increasing sample size, unfortunately you have diminishing returns on precision: As you can see in the dotted lines in Figure 2, the CI levels off, and you need disproportionally large sample sizes to squeeze out the last tiny percentages of a shrinking CI. If you follow Pareto’s principle, you should stop somewhen. Probably in scientific progress accuracy will be rather achieved in meta-analyzing several studies (which also gives you an estimate about the ES variability and possible moderators) than doing one mega-study.

**Hence: Always report your ES estimate, even in small- n studies.**

Kelley, K., & Maxwell, S. E. (2003). Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant. *Psychological Methods, 8*, 305–321. doi:10.1037/1082-989X.8.3.305

Kelley, K., & Rausch, J. R. (2006). Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals. *Psychological Methods, 11*, 363–385. doi:10.1037/1082-989X.11.4.363

Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation. *Annual Review of Psychology, 59(1)*, 537–563. doi:10.1146/annurev.psych.59.103006.093735

Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. *Review of General Psychology, 7*, 331–363. doi:10.1037/1089-2680.7.4.331

van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014). Why publishing everything is more effective than selective publishing of statistically significant results.* PLoS ONE, 9*, e84896. doi:10.1371/journal.pone.0084896

In order to approach this criticism it has been suggested to do a sensitivity analysis (or robustness analysis), that demonstrates how the choice of priors affects the conclusions drawn from the Bayesian analysis. Ideally, it can be shown that for virtually any reasonable prior the conclusions remain the same. In this case, the critique “it’s all in the prior” can be refuted on empirical grounds.

In their recent paper “The p < .05 rule and the hidden costs of the free lunch in inference” Jeff Rouder and colleagues argue that in the case of the default Bayes factor for *t* tests the choice of the H1 prior distribution does not make a strong difference (see Figure 6, right panel). They come to the conclusion that “Prior scale does matter, and may change the Bayes factor by a factor of 2 or so, but it does not change the order of magnitude.” (p. 24).

The default Bayes factor for *t* tests (Rouder, Speckman, Sun, Morey, Iverson, 2009) assumes that effect sizes (expressed as Cohen’s *d*) are distributed as a Cauchy distribution (this is the prior distribution for H1). The spread of the Cauchy distribution can be changed with the scale parameter *r*. Depending on the specific research area, one can use a wider (large *r*‘s, e.g. *r* =1.5) or a thinner (small r’s, e.g. r = 0.5) Cauchy distribution. This corresponds to the prior belief that typically larger or smaller effect sizes can be expected.

For the two-sample *t* test, the BayesFactor package for R suggest three defaults for the scale parameter:

- “medium” (r = sqrt(2)/2 = 0.71),
- “wide” (r = 1), and
- “ultra-wide” (r = sqrt(2) = 1.41).

Here’s a display for these distributions:

**For a given effect size: How does the choice of the prior distribution change the resulting Bayes factor?**

The following shiny app demonstrates how the choice of the prior influences the Bayes factor for a given effect size and sample size. Try moving the sliders! You can also provide arbitrary values for r (as comma-separated values; r must be > 0; reasonable ranges are between 0.2 and 2).

For a robustness analysis simply compare the lines at each vertical cut. An important line is the solid blue line at log(1), which indicates the same support for H1 and H0. All values above that line are in favor of the H1, all values below that line are in favor of H0.

As you will see, in most situations the Bayes factors for all *r*‘s are either above log(1), or below log(1). That means, regardless of the choice of the prior you will come to the same conclusion. There are very few cases where a data line for one *r* is below log(1) and the other is above log(1). In this case, different *r*‘s would come to different conclusions. But in these ambiguous situations the evidence for H1 or for H0 is always in the “anectodal” region, which is a very weak evidence. With the default r’s, the ratio of the resulting Bayes factors is indeed maximally “a factor of 2 or so”.

To summarize, within a reasonable range of prior distributions it is **not** possible that one prior generates strong evidence for H1, while some other prior generates strong evidence for H0. In that sense, the conclusions drawn from a default Bayes factor are robust to the choice of (reasonable) priors.

Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (submitted). The p < .05 rule and the hidden costs of the free lunch in inference. Retrieved from http://pcl.missouri.edu/biblio/author/29

Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin & Review, 16*, 225–237.

**[Update Oct 2014: Due to some changes to the Bayes factor calculator webpage, and as I understand BFs much better now, this post has been updated ...]**

I started to familiarize myself with Bayesian statistics. In this post I’ll show some insights I had about *Bayes factors* (BF).

Bayes factors provide a numerical value that quantifies how well a hypothesis predicts the empirical data relative to a competing hypothesis. For example, if the BF is 4, this indicates: “This empirical data is 4 times more probable if H₁ were true than if H₀ were true.” . Hence, evidence points towards H₁. A BF of 1 means that data are equally likely to be occured under both hypotheses.

More formally, the BF can be written as:

where D is the data. Hence, the BF is a ratio of probabilities, and is related to larger class of likelihood-ratio test.

What researchers usually are interested in is not p(Data | Hypothesis), but rather p(Hypothesis | Data). Using Bayes’ theorem, the former can be transformed into the latter by assuming prior probabilities for the hypotheses. The BF then tells one how to update one’s prior probabilities after having seen the data, using this formula (Berger, 2006):

Given a BF of 1, one does not have to update his or her priors. If one holds, for example, equal priors (p(H1) = p(H0) = .5), these probabilities do not change after having seen the data of the original study.

The best detailed introduction of BFs I know of can be be found in Richard Morey’s blog posts [1] [2][3]. Also helpful is the ever-growing tutorial page for the BayesFactor package. (For other introductions to BFs, see Wikipedia, Bayesian-Inference, the classic paper by Kass and Raftery, 1995, or Berger, 2006).

Although many authors agree about the many theoretical advantages of BFs, until recently it was complicated and unclear how to compute a BF even for the simple standard designs (Rouder, Morey, Speckman, & Province, 2012). Fortunately, over the last years *default Bayes factors* for several standard designs have been developed (Rouder et al., 2012; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Morey & Rouder, 2011). For example, for a two-sample *t* test, a BF can be derived simply by plugging the *t* value and the sample sizes into a formula. The BF is easy to compute by the R package BayesFactor (Morey & Rouder, 2013), or by online calculators [1][2].

When I started to familiarize myself with BFs, I was sometimes confused, as the same number seemed to mean different things in different publications. And indeed, **four types of Bayes factors** can be distinguished. “Under the hood”, all four types are identical, but you have to be aware which type has been employed in the specific case.

The first distinction is, whether the BF indicates “ over ” (=), or “ over ” (=). A of 2 means “Data is 2 times more likely to be occured under than under “, while the same situation would be a of 0.5 (i.e., the reciprocal value 1/2). Intuitively, I prefer larger values to be “better”, and as I usually would like to have evidence for instead of , I usually prefer the . However, if your goal is to show evidence *for* the H0, then is easier to communicate (compare: “Data occured 0.1 more likely under the alternative” vs. “Data show 10 times more evidence for the null than for the alternative”).

The second distinction is, whether one reports the raw BF, or the natural logarithm of the BF (The log(BF) has also been called “*weight of evidence*“; Good, 1985). The logarithm has the advantage that the scale above 1 (evidence for ) is identical to the scale below 1 (evidence for ). In the previous example, a of 2 is equivalent to a of 0.5. Taking the log of both values leads to = 0.69 and = -0.69: Same value, reversed sign. This makes the log(BF) ideal for visualization, as the scale is linear in both directions. Following graphic shows the relationship between raw/log BF:

As you can see in the Table of Figure 1, different authors use different flavors. This often makes sense, as we sometimes want to communicate evidence for the H1, and sometimes for the H0. However, for the uninitiated it can be sometimes confusing.

Usually, tables in publication report the raw BF (raw- or raw+). Plots, in contrast, typically use the log scale, for example:

Figure 2 shows conversion paths of the different BF flavors:

The user interface functions of the BayesFactor package always print the raw . Internally, however, the BF is stored as log().

Hence, you have to be careful when you directly use the backend utility functions, such as [cci]ttest.tstat[/cci]. These functions return the log(). As the conversion table shows, you have to [cci]exp()[/cci] that number to get the raw BF. Check the documentation of the functions if you are unsure which flavor is returned.

Related posts: Exploring the robustness of Bayes Factors: A convenient plotting function

Berger, J. O. (2006). Bayes factors. In S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic, & N. L. Johnson (Eds.), *Encyclopedia of Statistical Sciences, vol. 1 (2nd ed.)* (pp. 378–386). Hoboken, NJ: Wiley.

Good, I. J. (1985). Weight of evidence: A brief survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), *Bayesian Statistics 2* (pp. 249–270). Elsevier.

Morey, R. D. & Rouder, J. N. (2011). Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16(4), 406–419. PMID: 21787084. doi:10.1037/a0024377

Morey, R. D. & Rouder, J. N. (2013). {BAYESFACTOR}: computation of bayes factors for common designs. R package version 0.9.4. Retrieved from http://CRAN.R- project.org/package=BayesFactor

Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56(5), 356–374. doi:10.1016/j.jmp.2012.08.001

Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237.

Today a new version (0.23.1) of the WRS package (Wilcox’ Robust Statistics) has been released. This package is the companion to his rather exhaustive book on robust statistics, “Introduction to Robust Estimation and Hypothesis Testing” (Amazon Link de/us).

For a fail-safe installation of the package, follow this instruction.

As a guest post, Rand Wilcox describes the new functions of the newest WRS version – have fun!

“Hi everyone,

As you probably know, when standard assumptions are violated, classic methods for comparing groups and studying associations can have very poor power and yield highly misleading results. The better known methods for dealing with these problem (transforming the data or testing assumptions) are ineffective compared to more modern methods. Simply removing outliers among the dependent variable and applying standard techniques to the remaining data is disastrous.

Methods I derive to deal with these problems can be applied with R functions stored in an R package (WRS) maintained by Felix Schönbrodt. Felix asked me to briefly describe my recent efforts for a newsletter he posts. In case this might help some of you, a brief description of my recently developed methods and corresponding R functions are provided below. (The papers I cite illustrate that they can make a substantial difference compared to extant techniques.)

Sometimes it can be important and more informative to compare the tails (upper and lower quantiles) of two groups rather than a measure of location that is centrally located. Example: have been involved in a study aimed at determining whether intervention reduced depressive symptoms. But the typical individual was not very depressed prior to intervention and no difference is found using the more obvious techniques. Simply ignoring the less depressed individuals results in using the wrong standard error – a very serious problem. But comparing quantiles, it was found that the more depressed individuals benefitted the most from intervention.

The new method beats the shift function. See

Wilcox, R. R., Erceg-Hurn, D., Clark, F. & Carlson, M. (2013). Comparing two independent groups via the lower and upper quantiles. Journal of Statistical Computation and Simulation. DOI: 10.1080/00949655.2012.754026

Use the R function [cci]qcomhd[/cci].

For **dependent groups** must use another method. There are, in fact, two distinct ways of viewing the problem. See

Wilcox, R. R. & Erceg-Hurn, D. (2012). Comparing two dependent groups via quantiles. Journal of Applied Statistics, 39, 2655–2664.

Use the R function [cci]Dqcomhd[/cci].

When comparing two groups based on a Likert scale, use the function [cci]disc2com[/cci].

It performs a global test of P(X=x)=P(Y=x) for all x using a generalization of the Storer–Kim method for comparing binomials.

[cci]binband[/cci]: a multiple comparison method for the individual cell probabilities.

[cci]tshdreg[/cci]: This is a modification of the Theil–Sen estimator. When there are tied values among the dependent variable, this modification might result in substantially higher power. A paper (Wilcox & Clark, in press) provides details. The function [cci]tsreg[/cci] now checks whether there are any tied values and prints a message suggesting that you might want to use [cci]tshdreg[/cci] instead.

[cci]qhdsm[/cci]: A quantile regression smoother. That is, plot the regression line when predicting some quantile without specifying a parametric form for the regression line. Multiple quantile regression lines can be plotted. The method can be more satisfactory than using the function [cci]qsmcobs[/cci] (a spline-type method), which often creates waves and curvature that give an incorrect sense of the association. Another advantage of [cci]qhsdm[/cci] is that it can be used with more than one predictor; [cci]qsmcobs[/cci] is limited to one predictor only. The strategy behind [cci]qhdsm[/cci] is to get an initial approximation of the regression line using a running interval smoother in conjunction with the Harrell–Davis quantile estimator and then smoothed again via LOESS.

It is surprising how often an association is found when dealing with the higher and lower quantiles of the dependent variable that are not detected by least squares and other robust estimators.

[cci]qhdsm2g[/cci]: Plots regression lines for two groups using the function [cci]qhdsm[/cci].

[cci]rplot[/cci] has been updated: setting the argument [cci]LP=TRUE[/cci] gives a smoother regression line.

[cci]rplotCI[/cci]. Same as [cci]rplot[/cci] but includes lines indicating a confidence interval for the predicted Y values

[cci]rplotpbCI[/cci]. Same as [cci]rplotCI[/cci], only use a bootstrap method to compute confidence intervals.

[cci]ancJN[/cci]: The function fits a robust regression line for each group and then determines whether the predicted Y values differ significantly at specified points. So it has connections to the classic Johnson-Neyman method. That is, the method provides an indication of where the regression lines cross. Both types of heteroscedasticity are allowed, which can result in improved power beyond the improved power stemming from a robust estimator. See

Wilcox, R. R. (2013). A heteroscedastic method for comparing regression lines at specified design points when using a robust regression estimator. Journal of Data Science, 11, 281–291

[cci]anctspb[/cci]: Like [cci]ancJN[/cci] but uses a percentile bootstrap method that might help when there are tied values among the dependent variable.

[cci]ancGLOB[/cci]. A robust global ANCOVA method. Like the function ancova, it provides a flexible way of dealing with curvature and heteroscedasticity is allowed. But this function can reject in situations where ancova does not reject. The function returns a p-value and the hypothesis of identical regression lines is rejected if the p-value is less than or equal to a critical p-value. In essence, it can beat reliance on improved versions of the Bonferroni method. (Details are in a paper submitted for publication.) It does not dominate my original ANCOVA method (applied with the R function [cci]ancova[/cci]) in terms of power, but have encountered situations where it makes a practical difference.

It determines a critical p-value via the R function [cci]ancGLOB_pv[/cci].

In essence, simulations are used. By default, the number of replications is [cci]iter=500[/cci]. But suggest using [cci]iter=2000[/cci] or larger. Execution time can be reduced substantially with [cci]cpp=TRUE[/cci], which calls a C++ version of the function written by Xiao He. Here are the commands to install the C++ version:

[cc lang="rsplus" escaped="true"]

install.packages(‘devtools’)

library(‘devtools’)

install_github(‘WRScpp’, ‘mrxiaohe’)

[/cc]

For a global test that two parametric regression lines are identical, see

Wilcox, R. R. & Clark, F. (in press). Heteroscedastic global tests that the regression parameters for two or more independent groups are identical. Communications in Statistics– Simulation and Computation.

[cci]ancGpar[/cci] performs the robust method. The paper includes a different method when using least squares regression. It is based in part on the HC4 estimator, which deals with heteroscedasticity. But if there are outliers among the dependent variable, you are much better off using a robust estimator.

[cci]Dancova[/cci]: ANCOVA for two dependent groups that provides a flexible way of dealing with curvature. Both types of heteroscedasticity are allowed. Roughly, approximate the regression lines with a running interval smoother and at specified design points compare the regression lines. This is an extension of the R function [cci]ancova[/cci] to dependent groups. The function can do an analysis on either the marginal measures of location or a measure of location based on the difference scores. When using a robust estimator, the choice between these two approaches can be important. Defaults to using a trimmed mean.

[cci]Dancovamp[/cci]: Like [cci]Dancova[/cci] only designed to handle multiple covariates.

[cci]Danctspb[/cci]: Compare regression lines of two dependent groups using a robust regression estimator. The default is to use Theil–Sen, but any estimator can be used via the argument regfun. So in contrast to [cci]Dancova[/cci], a parametric form for the regression line is made. As usual, can eliminate outliers among the independent variable by setting the argument [cci]xout=TRUE[/cci]. When a parametric regression line provides a more accurate fit, can have more power compared to using a smoother. But when there is curvature that is not modeled well with a parametric fit, the reverse can happen.

Note: a version of [cci]ancGLOB[/cci] for dependent groups is being studied.

[cci]Rcoefalpha[/cci]: computes a robust analog of coefficient alpha. Developed this method some years ago but just got around to writing an R function. See

Wilcox, R. R. (1992). Robust generalizations of classical test reliability and Cronbach’s alpha. British Journal of Mathematical and Statistical Psychology, 45, 239–254.

*R. R. Wilcox*”

Have fun exploring these new methods!

]]>In the uncommon situation that the data are extensive and of simple structure, the prior assumptions will be unimportant and the assumed sampling model will be uncontroversial. More generally we would like to report that any conclusions are robust to reasonable changes in both prior and assumed model: this has been termed inference robustness

(David Spiegelhalter and Kenneth Rice (2009) Bayesian statistics. Scholarpedia, 4(8):5230.)

Therefore, it is suggested that …

In particular, audiences should ideally fully understand the contribution of the prior distribution to the conclusions. (ibid)

In the example of Bayes factors for t tests (Rouder, Speckman, Sun, Morey, & Iverson, 2009), the assumption that has to be defined a priori is the effect size δ expected under the H1. In the BayesFactor package for R, this can be adjusted via the r parameter. By default, it is set to 0.5, but it can be made wider (larger r’s, which means one expects larger effects) or narrower (r’s close to zero, which means one expects smaller effects in the population).

In their reanalysis of Bem’s ESP data, Wagenmakers, Wetzels, Borsboom, Kievit, and van der Maas (2011, PDF) proposed a robustness analysis for Bayes factors (BF), which simply shows the BF for a range of priors. If the conclusion is the same for a large range of priors, it could be judged to be robust (this is also called a “sensitivity analysis”).

I wrote an R function that can generate plots like this. Here’s a reproduction of Wagenmakers’ et al (2011) analysis of Bem’s data – it looks pretty identical:

[cc lang="rsplus" escaped="true"]

## Bem data, two sided

# provide t values, sample sizes, and the location(s) of the red dot(s)

# set forH1 to FALSE in order to vertically flip the plot.

# Usually I prefer higher BF to be in favor of H1, but I flipped it in order to match Wagenmakers et al (2011)

BFrobustplot(

ts=c(2.51, 2.55, 2.23, 1.74, 1.92, 2.39, 2.03, 1.8, 1.31, 2.96),

ns=c(100, 97, 100, 150, 100, 150, 99, 150, 200, 50),

dots=1, forH1 = FALSE)

[/cc]

You can throw in as many t values and corresponding sample sizes as you want. Furthermore, the function can compute one-sided Bayes factors as described in Wagenmakers and Morey (2013). If this approach is applied to the Bem data, the plot looks as following – everything is shifted a bit into the H1 direction:

[cc lang="rsplus" escaped="true"]

# Bem data one-sided

BFrobustplot(

ts=c(2.51, 2.55, 2.23, 1.74, 1.92, 2.39, 2.03, 1.8, 1.31, 2.96),

ns=c(100, 97, 100, 150, 100, 150, 99, 150, 200, 50),

dots=1, sides=”one”, forH1 = FALSE)

[/cc]

Finally, here’s the function:

[cc lang="rsplus" escaped="true"]

## This source code is licensed under the FreeBSD license

## (c) 2013 Felix Schönbrodt

#’ @title Plots a comparison of a sequence of priors for t test Bayes factors

#’

#’ @details

#’

#’

#’ @param ts A vector of t values

#’ @param ns A vector of corresponding sample sizes

#’ @param rs The sequence of rs that should be tested. r should run up to 2 (higher values are implausible; E.-J. Wagenmakers, personal communication, Aug 22, 2013)

#’ @param labels Names for the studies (displayed in the facet headings)

#’ @param dots Values of r’s which should be marked with a red dot

#’ @param plot If TRUE, a ggplot is returned. If false, a data frame with the computed Bayes factors is returned

#’ @param sides If set to “two” (default), a two-sided Bayes factor is computed. If set to “one”, a one-sided Bayes factor is computed. In this case, it is assumed that positive t values correspond to results in the predicted direction and negative t values to results in the unpredicted direction. For details, see Wagenmakers, E. J., & Morey, R. D. (2013). Simple relation between one-sided and two-sided Bayesian point-null hypothesis tests.

#’ @param nrow Number of rows of the faceted plot.

#’ @param forH1 Defines the direction of the BF. If forH1 is TRUE, BF > 1 speak in favor of H1 (i.e., the quotient is defined as H1/H0). If forH1 is FALSE, it’s the reverse direction.

#’

#’ @references

#’

#’ Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin and Review, 16, 225-237.

#’ Wagenmakers, E.-J., & Morey, R. D. (2013). Simple relation between one-sided and two-sided Bayesian point-null hypothesis tests. Manuscript submitted for publication

#’ Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Kievit, R. & van der Maas, H. L. J. (2011). Yes, psychologists must change the way they analyze their data: Clarifications for Bem, Utts, & Johnson (2011)

BFrobustplot <- function(

ts, ns, rs=seq(0, 2, length.out=200), dots=1, plot=TRUE,

labels=c(), sides="two", nrow=2, xticks=3, forH1=TRUE)

{

library(BayesFactor)

# compute one-sided p-values from ts and ns

ps <- pt(ts, df=ns-1, lower.tail = FALSE) # one-sided test

# add the dots location to the sequences of r's

rs <- c(rs, dots)

res <- data.frame()

for (r in rs) {

# first: calculate two-sided BF

B_e0 <- c()

for (i in 1:length(ts))

B_e0 <- c(B_e0, exp(ttest.tstat(t = ts[i], n1 = ns[i], rscale=r)$bf))

# second: calculate one-sided BF

B_r0 <- c()

for (i in 1:length(ts)) {

if (ts[i] > 0) {

# correct direction

B_r0 <- c(B_r0, (2 - 2*ps[i])*B_e0[i])

} else {

# wrong direction

B_r0 <- c(B_r0, (1 - ps[i])*2*B_e0[i])

}

}

res0 <- data.frame(t=ts, n=ns, BF_two=B_e0, BF_one=B_r0, r=r)

if (length(labels) > 0) {

res0$labels <- labels

res0$heading <- factor(1:length(labels), labels=paste0(labels, "\n(t = ", ts, ", df = ", ns-1, ")"), ordered=TRUE)

} else {

res0$heading <- factor(1:length(ts), labels=paste0("t = ", ts, ", df = ", ns-1), ordered=TRUE)

}

res <- rbind(res, res0)

}

# define the measure to be plotted: one- or two-sided?

res$BF <- res[, paste0("BF_", sides)]

# Flip BF if requested

if (forH1 == FALSE) {

res$BF <- 1/res$BF

}

if (plot==TRUE) {

library(ggplot2)

p1 <- ggplot(res, aes(x=r, y=log(BF))) + geom_line() + facet_wrap(~heading, nrow=nrow) + theme_bw() + ylab("log(BF)")

p1 <- p1 + geom_hline(yintercept=c(c(-log(c(30, 10, 3)), log(c(3, 10, 30)))), linetype="dotted", color="darkgrey")

p1 <- p1 + geom_hline(yintercept=log(1), linetype="dashed", color="darkgreen")

# add the dots

p1 <- p1 + geom_point(data=res[res$r %in% dots,], aes(x=r, y=log(BF)), color="red", size=2)

# add annotation

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=-2.85, label=paste0("Strong~H[", ifelse(forH1==TRUE,0,1), "]"), hjust=1, vjust=.5, size=3, color="black", parse=TRUE)

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=-1.7 , label=paste0("Moderate~H[", ifelse(forH1==TRUE,0,1), "]"), hjust=1, vjust=.5, size=3, color="black", parse=TRUE)

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=-.55 , label=paste0("Anectodal~H[", ifelse(forH1==TRUE,0,1), "]"), hjust=1, vjust=.5, size=3, color="black", parse=TRUE)

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=2.86 , label=paste0("Strong~H[", ifelse(forH1==TRUE,1,0), "]"), hjust=1, vjust=.5, size=3, color="black", parse=TRUE)

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=1.7 , label=paste0("Moderate~H[", ifelse(forH1==TRUE,1,0), "]"), hjust=1, vjust=.5, size=3, color="black", parse=TRUE)

p1 <- p1 + annotate("text", x=max(rs)*1.8, y=.55 , label=paste0("Anectodal~H[", ifelse(forH1==TRUE,1,0), "]"), hjust=1, vjust=.5, vjust=.5, size=3, color="black", parse=TRUE)

# set scale ticks

p1 <- p1 + scale_y_continuous(breaks=c(c(-log(c(30, 10, 3)), 0, log(c(3, 10, 30)))), labels=c("-log(30)", "-log(10)", "-log(3)", "log(1)", "log(3)", "log(10)", "log(30)"))

p1 <- p1 + scale_x_continuous(breaks=seq(min(rs), max(rs), length.out=xticks))

return(p1)

} else {

return(res)

}

}

[/cc]

*References*

Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin and Review, 16*, 225-237. [for a PDF, see bottom of this page]

Wagenmakers, E.-J., & Morey, R. D. (2013). Simple relation between one-sided and two-sided Bayesian point-null hypothesis tests. Manuscript submitted for publication (website)

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Kievit, R. & van der Maas, H. L. J. (2011). Yes, psychologists must change the way they analyze their data: Clarifications for Bem, Utts, & Johnson (2011) [PDF]

The guys from RStudio now provide CRAN download logs (see also this blog post). Great work!

I always asked myself, how many people actually download my packages. Now I finally can get an answer (… with some anxiety to get frustrated

Here are the complete, self-contained R scripts to analyze these log data:

[cc lang="rsplus" escaped="true"]## ======================================================================

## Step 1: Download all log files

## ======================================================================

# Here’s an easy way to get all the URLs in R

start <- as.Date('2012-10-01')

today <- as.Date('2013-06-10')

all_days <- seq(start, today, by = 'day')

year <- as.POSIXlt(all_days)$year + 1900

urls <- paste0('http://cran-logs.rstudio.com/', year, '/', all_days, '.csv.gz')

# only download the files you don't have:

missing_days <- setdiff(as.character(all_days), tools::file_path_sans_ext(dir("CRANlogs"), TRUE))

dir.create("CRANlogs")

for (i in 1:length(missing_days)) {

print(paste0(i, "/", length(missing_days)))

download.file(urls[i], paste0('CRANlogs/', missing_days[i], '.csv.gz'))

}

[/cc]

[cc lang="rsplus" escaped="true"]## ======================================================================

## Step 2: Load single data files into one big data.table

## ======================================================================

file_list <- list.files("CRANlogs", full.names=TRUE)

logs <- list()

for (file in file_list) {

print(paste("Reading", file, "..."))

logs[[file]] <- read.table(file, header = TRUE, sep = ",", quote = "\"",

dec = ".", fill = TRUE, comment.char = "", as.is=TRUE)

}

# rbind together all files

library(data.table)

dat <- rbindlist(logs)

# add some keys and define variable types

dat[, date:=as.Date(date)]

dat[, package:=factor(package)]

dat[, country:=factor(country)]

dat[, weekday:=weekdays(date)]

dat[, week:=strftime(as.POSIXlt(date),format="%Y-%W")]

setkey(dat, package, date, week, country)

save(dat, file="CRANlogs/CRANlogs.RData")

# for later analyses: load the saved data.table

# load("CRANlogs/CRANlogs.RData")

[/cc]

[cc lang="rsplus" escaped="true"]## ======================================================================

## Step 3: Analyze it!

## ======================================================================

library(ggplot2)

library(plyr)

str(dat)

# Overall downloads of packages

d1 <- dat[, length(week), by=package]

d1 <- d1[order(V1), ]

d1[package=="TripleR", ]

d1[package=="psych", ]

# plot 1: Compare downloads of selected packages on a weekly basis

agg1 <- dat[J(c("TripleR", "RSA")), length(unique(ip_id)), by=c("week", "package")]

ggplot(agg1, aes(x=week, y=V1, color=package, group=package)) + geom_line() + ylab("Downloads") + theme_bw() + theme(axis.text.x = element_text(angle=90, size=8, vjust=0.5))

agg1 <- dat[J(c("psych", "TripleR", "RSA")), length(unique(ip_id)), by=c("week", "package")]

ggplot(agg1, aes(x=week, y=V1, color=package, group=package)) + geom_line() + ylab("Downloads") + theme_bw() + theme(axis.text.x = element_text(angle=90, size=8, vjust=0.5))[/cc]

Here are my two packages, `TripleR`

and `RSA`

. Actually, ~30 downloads per week (from this single mirror) is much more than I’ve expected!

To put things in perspective: package `psych`

included in the plot:

Some psychological sidenotes on social comparisons:

- Downward comparisons enhance well-being, extreme upward comparisons are detrimental. Hence, do
*never*include`ggplot2`

into your graphic! - Upward comparisons instigate your achievement motive, and give you drive to get better. Hence, select some packages, which are slightly above your own.
- Of course, things are a bit more complicated than that …

*All source code on this post is licensed under the FreeBSD license.*