This is a guest post from Felix Suessenbach.
Researchers of motivational psychology have long struggled with the power motive’s heterogeneous definition encompassing elements such as desires for dominance, reputation, prestige, leadership, and status (e.g., Winter, 1973). This heterogeneity has likely been responsible for researchers having found different relationships between the power motive and external variables depending on which power motive scale they used (e.g., Engeser & Langens, 2010). Thus, to provide a long-needed taxonomy of clearly distinguishable power motive components we developed the dominance, prestige, and leadership (DoPL) account of social power motives. In particular we differentiate between:
The dominance motive, defined as a desire for coercive power obtained through threats, intimidation, or deception
The prestige motive, defined as a desire for voluntary deference obtained through others’ admiration and respect particularly for one’s valued skills and knowledge
The leadership motive, defined as a desire for legitimised power granted by one’s group and obtained through taking responsibility in and for this group
Opposed to previous attempts to differentiate different power motive components (e.g., socialised and personalised power; McClelland, 1970) the DoPL account of social power motives is based on a solid theoretical framework adapted from research into social hierarchies (e.g., Cheng, Tracy, & Henrich, 2010; Henrich & Gil-White, 2001). Thus, the DoPL account does not suffer from strongly different interpretations of how these components manifest themselves.
Using newly developed DoPL questionnaires we showed the DoPL motives can be measured both reliably and distinctively (study 1). Moreover, we showed these DoPL motives strongly related to a common power desire (study 2), explaining more than 80% of variance in two established power motive scales (UMS power, Schönbrodt & Gerstenberg, 2012; PRF dominance, Jackson, 1984). Assessing their nomological networks (studies 3 & 4), we demonstrated distinct associations such as between…
the dominance motive and self-reported anger and verbal aggression
the prestige motive and self-reported fear of losing reputation and claiming to have higher moral concerns
the leadership motive and self-reported emotional stability and helping behaviour
Regarding observed behaviour and other external variables (studies 5 to 7) we found:
The dominance motive uniquely and negatively predicted the amount of money given to another player in a dictator game after having received nothing in two previous dictator games. This effect can be explained by a combination of general agonistic tendencies as well as retaliatory desires related to the dominance motive.
The leadership motive uniquely predicted the attainment of higher employment ranks across all kinds of professions. This effect was somewhat stronger in females which might be explained by discrimination against females regarding promotions and thus females having to compensate by being more highly motivated to reach high leadership positions.
When donating behaviour to a charity was made overt, residualised dominance motives (i.e., controlled for shared prestige and leadership influences) related negatively to the overall proportion donated to a charity as well as the probability to donate. Whereas residualised leadership motives only related positively to the overall amount donated to charity, residualised prestige motives only related positively to the probability to donate. Thus, to some degree, dominance desires relate negatively and leadership and prestige desires positively to prosocial donating behaviour.
This research shows that different power motive components in many (but not all) cases relate differently to a range of external variables. Thus, to improve the prediction of influential power-relevant behaviour as a function of individuals’ power desires we invite researchers to employ this novel taxonomy of power motives to further advance this important field of research.
Cheng, J. T., Tracy, J. L., & Henrich, J. (2010). Pride, personality, and the evolutionary foundations of human social status. Evolution and Human Behavior, 31, 334–347 https://doi.org/10.1016/
Engeser, S., & Langens, T. (2010). Mapping explicit social motives of achievement, power, and affiliation onto the five-factor model of personality. Scandinavian Journal of Psychology, 51, 309–318 https://doi.org/10.1111/j.1467-9450.2009.00773.x.
Henrich, J., & Gil-White, F. J. (2001). The evolution of prestige: Freely conferred deference as a mechanism for enhancing the benefits of cultural transmission. Evolution and Human Behavior, 22, 165–196 https://doi.org/10.1016/S1090-5138(00)00071-4.
Jackson, D. N. (1984). Personality research form manual (3rd ed.). Port Huron: Research Psychologists Press.
McClelland, D. C. (1970). The two faces of power. Journal of International Affairs, 24, 29–47.
Schönbrodt, F. D., & Gerstenberg, F. X. R. (2012). An IRT analysis of motive questionnaires: The unified motive scales. Journal of Research in Personality, 46, 725–742 https://doi.org/10.1016/j.jrp.2012.08.010. [Free PDF on OSF]
Winter, D. G. (1973). The power motive. New York: The Free Press.
Our department embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open materials, and study pre-registration. Candidates are asked to describe in what way they already pursued and plan to pursue these goals.
Since then, every professorship announcement contained this paragraph (and we made good experiences with it).
I am very happy to announce that my department now turned this implicit policy into an explicit hiring policy, effective since May 2018: The department’s steering committee unanimously voted for an explicit policy to always include this (or a similar) statement to all future professorship job advertisements.
It is the task of the appointment committee to value the existing open science activities as well as future commitments of applicants appropriately. By including this statement, our department aims to communicate core values of good scientific practice and to attract excellent researchers who aim for transparent and credible research.
In this respect, take a look at the current draft of a Modular Certification Initiative (initiated by Chris Chambers, Kyle MacDonald and me, with a lot of input from the open science community). With this TOP-like scheme, institutions, but also single researchers, can select a level of openness which they require in their hiring process.
So, if you want to join the LMU psychology department as a professor, you should better have some open science track record.
Previous investigations typically looked only at publication bias or questionable research practices QRPs (but not both), used non-representative study-level sample sizes, or only compared few bias-correcting techniques, but not all of them. Our goal was to simulate a research literature that is as realistic as possible for psychology. In order to simulate several research environments, we fully crossed five experimental factors: (1) the true underlying effect, δ (0, 0.2, 0.5, 0.8); (2) between-study heterogeneity, τ (0, 0.2, 0.4); (3) the number of studies in the meta-analytic sample, k (10, 30, 60, 100); (4) the percentage of studies in the meta-analytic sample produced under publication bias (0%, 60%, 90%); and (5) the use of QRPs in the literature that produced the meta-analytic sample (none, medium, high).
This blog post summarizes some insights from our study, internally called “meta-showdown”. Check out the preprint; and the interactive app metaExplorer. The fully reproducible and reusable simulation code is on Github, and more information is on OSF.
In this blog post, I will highlight some lessons that we learned during the project, primarily focusing on what not do to when performing a meta-analysis.
Constraints on Generality disclaimer: These recommendations apply to typical sample sizes, effect sizes, and heterogeneities in psychology; other research literatures might have different settings and therefore a different performance of the methods. Furthermore, the recommendations rely on the modeling assumptions of our simulation. We went a long way to make them as realistic as possible, but other assumptions could lead to other results.
If studies have no publication bias, nothing can beat plain old random effects meta-analysis: it has the highest power, the least bias, and the highest efficiency compared to all other methods. Even in the presence of some (though not extreme) QRPs, naive RE performs better than all other methods. When can we expect no publication bias? If (and, in my opinion only if) we meta-analyze a set of registered reports.
In any other setting except registered reports, a consequential amount of publication bias must be expected. In the field of psychology/psychiatry, more than 90% of all published hypothesis tests are significant (Fanelli, 2011) despite the average power being estimated as around 35% (Bakker, van Dijk, & Wicherts, 2012) – the gap points towards a huge publication bias. In the presence of publication bias, naive random effects meta-analysis and trim-and-fill have false positive rates approaching 100%:
More thoughts about trim-and-fill’s inability to recover δ=0 are in Joe Hilgard’s blog post. (Note: this insight is not really new and has been shown multiple times before, for example by Moreno et al., 2009, and Simonsohn, Nelson, and Simmons, 2014).
Our recommendation: Never trust meta-analyses based on naive random effects and trim-and-fill, unless you can rule out publication bias. Results from previously published meta-analyses based on these methods should be treated with a lot of skepticism.
Update 2017/06/09: We had a productive exchange with Uri Simonsohn and Joe Simmons concerning what should be estimated in a meta-analysis with heterogeneity. Traditionally, meta-analysts have tried to arrive at techniques that recover the true average effect of all conducted studies (AEA – average effect of all studies). Simonsohn et al (2014) propose estimating a different magnitude; the average effect of the studies one observes, rather than of all studies (AEO – average effect of observed studies). See Simonsohn et al (2014), the associated Supplementary Material 2, and also this blog post for arguments why they think this is a more useful quantity to estimate.
Hence, an investigation of the topic can be done on two levels: A) What is the more appropriate estimand (AEA or AEO?), and B) Under what conditions are estimators able to recover the respective true value with the least bias and least variance?
Instead of updating the section of the current blog post in the light of our discussion, I decided to cut it out and to move the topic to a future blog post. Likewise, one part of our manuscript’s revision will be a more detailed discussion about excatly these differences.
I archived the previous version of the blog post here.
Many bias-correcting methods are driven by QRPs – the more QRPs, the stronger the downward correction. However, this effect can get so strong, that methods overadjust into the opposite direction, even if all studies in the meta-analysis are of the same sign:
Note: You need to set the option “Keep negative estimates” to get this plot.
Our recommendation: Ignore bias-corrected results that go into the opposite direction; set the estimate to zero, do not reject H₀.
Typical small-study effects (e.g., by p-hacking or publication bias) induce a negative correlation between sample size and effect size – the smaller the sample, the larger the observed effect size. PET-PEESE aims to correct for that relationship. In the absence of bias and QRPs, however, random fluctuations can lead to a positive correlation between sample size and effect size, which leads to a PET and PEESE slope of the unintended sign. Without publication bias, this reversal of the slope actually happens quite often.
See for example the next figure. The true effect size is zero (red dot), naive random effects meta-analysis slightly overestimates the true effect (see black dotted triangle), but PET and PEESE massively overadjust towards more positive effects:
As far as I know, PET-PEESE is typically not intended to correct in the reverse direction. An underlying biasing process would have to systematically remove small studies that show a significant result with larger effect sizes, and keep small studies with non-significant results. In the current incentive structure of psychological research, I see no reason for such a process, unless researchers are motivated to show that a (maybe politically controversial) effect does not exist.
Our recommendation: Ignore the PET-PEESE correction if it has the wrong sign, unless there are good reasons for an untypical selection process.
A bias can be more easily accepted if it always is conservative – then one could conclude: “This method might miss some true effects, but if it indicates an effect, we can be quite confident that it really exists”. Depending on the conditions (i.e., how much publication bias, how much QRPs, etc.), however, PET/PEESE sometimes shows huge overestimation and sometimes huge underestimation.
For example, with no publication bias, some heterogeneity (τ=0.2), and severe QRPs, PET/PEESE underestimates the true effect of δ = 0.5:
In contrast, if no effect exists in reality, but strong publication bias, large heterogeneity and no QRPs, PET/PEESE overestimates at lot:
In fact, the distribution of PET/PEESE estimates looks virtually identical for these two examples, although the underlying true effect is δ = 0.5 in the upper plot and δ = 0 in the lower plot. Furthermore, note the huge spread of PET/PEESE estimates (the error bars visualize the 95% quantiles of all simulated replications): Any single PET/PEESE estimate can be very far off.
Our recommendation: As one cannot know the condition of reality, it is probably safest not to use PET/PEESE at all.
Again, please consider the “Constraints on Generality” disclaimer above.
As with any general recommendations, there might be good reasons to ignore them.