In our current draft (not public yet) we want to decenter (A) scientific publications as the primary research output that counts, and recommend to also take (B) published data sets, and the development and maintenance of (C) research software into consideration. (Along with Recognition and Rewards and other initiatives, we also call for taking Teaching, Leadership skills, Service to the institution/field, and Societal impact into account. In the white paper, however, we only address the operationalization of the Research dimension).
Concerning research software, we worked on an operationalization. This is inspired from:
Please note that …
Here is our current draft of the research software section. As we are not aware of any concrete implementation of assessing research software for hiring or promotion purposes (at least not in psychology or neighboring fields), we like to ask the community for feedback. At the end of the post we list three ways how you can comment.
Research software is a vital part of modern data-driven science that fuels both data collection (e.g., PsychoPy, Peirce et al., 2019, or lab.js, Henninger et al., 2021) and analysis (see, for example, R and the many contributed packages). In some cases, the functioning of entire scientific disciplines depends on the work of a few (often unpaid) software maintainers of critical software (Muna et al., 2016). Furthermore, non-commercial open source software is a necessary building block for computational transparency, reproducibility, and a thriving and inclusive scientific community. Instead of being a “career suicide”, it is high time that research software development is properly acknowledged in hiring and promotion.
Some research software is accompanied with a citable paper describing the software (e.g., for the lavaan structural equation modeling package in R: Rosseel, 2012). However, these “one-shot” descriptions of software often do not appropriately reflect the continuous work and changing teams that are necessary to develop and maintain research software. Therefore we include “Contributions to Research Software” as a separate category with their own quality criteria. Note that this category (C) only refers to dedicated, reusable research software, not to specific analysis scripts for a particular project. The latter should be listed under “Open reproducible scripts” of the respective paper in section (A).
For the evaluation of contributed research software, applicants can list up to 5 software artifacts along with the self-assessment criteria presented in Table 3 (a more comprehensive evaluation scheme with more quality criteria is proposed in Appendix A). Contributor roles are taken from the INRIA Evaluation Committee Criteria for Software Self-Assessment.
Table 3. Simple evaluation scheme for research software, with one specific example
Research Software 1 | URL | Comment | |
Title | R package RSA | https://CRAN.R-project.org/package=RSA | |
Citation | Schönbrodt, F. D. & Humberg, S. (2021). RSA: An R package for response surface analysis (version 0.10.4). Retrieved from https://cran.r-project.org/package=RSA | ||
Short description | An R package for Response Surface Analysis | ||
Date of first full release | 2013 | Necessary to compute citations relative to age of software | |
Date of most recent major release | 2020 | Indicates whether software is actively maintained | |
Contributor roles and involvement | DA-3 CD-3 MS-3 | What has the applicant contributed? For each of the 3 roles: – design and architecture (DA) – coding and debugging (CD) – maintenance and support (MS) … specify if you are: 0. not involved 1. an occasional contributor 2. a regular contributor 3. a main contributor Example: DA-2, CD-3, MS-1 | |
License | GPLv3 | Is the software open source? | |
Scientific impact indicators: | |||
Downloads or users per month | 710 downloads / month | https://cranlogs.r-pkg.org/badges/RSA | |
Citations | 110 | https://scholar.google.de/citations?view_op=view_citation&hl=de&user=KMy_6VIAAAAJ&citation_for_view=KMy_6VIAAAAJ:mB3voiENLucC | Evaluate relative to the age of software |
Other impact indicators (optional) | – | E.g., Github stars, number of dependencies. Be careful and responsible when using metrics, in particular when they are black-box algorithms. | |
Reusability indicator | R3 | Levels of the reusability indicator: R1 (0.25 points): Single scripts, loose documentation, no long-term maintenance. Prototype: A collection of reusable R scripts on OSF. R2 (1 points): Well-developed and tested software, fairly extensive documentation. Some attention to usability and user feedback. Not necessarily regularly updated. Prototype: A small CRAN package with no more active development (just maintenance) R3 (2 points): Major software project, strong attention to functionality and usability, extensive documentation, systematic bug chasing and unit testing, external quality control (e.g. by uploading to CRAN). Regularly updated. Prototype: Well received and actively maintained CRAN package. R4 (6 points): Critical infrastructure software. Hundreds of research projects use or depend on the software (+ all criteria of R3). Prototype: lavaan package. | |
Merit / impact statement (narrative, max 100 words) | The RSA package has become a standard package for computing and visualizing response surface analyses in psychology. A PsycInfo search for “response surface analysis” (from 2022-05-18) revealed that of the 20 most recent publications, 35% used our package (although 2 of 7 did not cite it). Several features, such as computation of multiple standard models and model comparisons are unique to this package. | ||
Reward Points | (3+3+3)/3 * 3 = 9 | Take the average value of the 3 contributor roles and multiply it with the points of the level of the reusability indicator. |
Is there essential information missing in the table?
We also want to offer a suggestion how to compute „reward points“. The goal is to bring the categories of „publications“ and „software contributions“ onto a common evaluative dimension. This gets a bit complicated, as we also propose bonus points for publications with certain quality criteria, so not every publication gets the same number of points. For the moment, imagine a publication of good quality (neither a quickly churned out low-quality publication, nor an outstanding, seminal contribution). What is the “paper equivalent” of a software contribution? Note that these bonus points are thought as incremental to an existing paper that describes the software.
Here’s our suggestion, being aware that it is easy to find counter-examples that do not fit in the system. But we are happy if our system is an incremental improvement over the status quo (which is: to ignore software contributions and to count the number of papers without any quality weighting):
Research Software Prototype | Paper equivalents (of good quality) |
---|---|
Simple script (a few hundred lines) with reuse potential, completely done by applicant | 0.25 |
A well-developed CRAN package: Occasional co-developer with a minor contribution | 0.5 |
A well-developed CRAN package: Active co-developer with major contribution | 1 |
A well-developed CRAN package: Main developer | 2 |
Critical infrastructure: Regular co-developer | 2 |
Critical infrastructure (e.g., lavaan): Main developer | 5 |
How to comment?
If you have comments, you can …
Thanks for your help!
]]>I want to invest my reviewing work in research that is worth to be reviewed. Furthermore, I do not want to further increase the billion dollar donations to premium publishers any more.
When deciding whether to accept or reject a review, I apply the following heuristics:
After these initial eligibility checks, I apply the following weights:
I anticipate that my criteria will gradually shift more and more to the top categories.
I realize that this personal policy has some side effects. For example, I really appreciate the good work of the editorial team from Nature Human Behavior. They did a lot to improve standards and policies at a Nature journal. So, while I’d be happy to support that specific team, I do not want to support SpringerNature as a profit organization; even more as they now test a scheme where they take a „a non-refundable fee of €2,190 to cover an editorial assessment and the peer-review process“ (Nature journals reveal terms of landmark open-access option). Wait – reviewers now get paid for their work? No, of course not. Researchers still do the reviews for free, as always. Nature now wants to get paid for your reviews.
I hope that with that reviewing policy I can make a small change towards a more open, more credible, and more efficient academic system. At least I feel much better with these priorities and have more fun reviewing.
[1] I employ the following heuristic: To keep the current academic system going, I have to review three papers for each paper that I submit as first author (including all revisions, as they usually require additional reviews). I clearly exceed this heuristic a lot.
]]>One criticism of such licenses stems from the definition of “freedom”: According to this point of view, the highest degree of freedom is if you can do anything with a material. This also includes commercial usage, which is usually closed for competitive reasons, or to integrate the material into a larger dataset which itself can not be open, because other parts of the data have restrictive licenses. We are not lawyers, but in our understanding this could, for example, also include restrictions due to privacy rights.
For example, imagine the compilation of an integrative database that includes both material from a copyleft source and another source that has individual-related material, which cannot be openly shared due to privacy rights (but could be shared as a restricted scientific use file). At least from our understanding, a strict copyleft license would preclude the reuse in such a restricted way. Hence, the copyleft license, although claiming to ensure freedom, does preclude a lot of potential reuse scenarios. From this point of view, a so-called permissive license (such as CC0, MIT, or BSD) provides more freedom than a copyleft license (see, e.g., The Whys and Hows of Licensing Scientific Code).
We propose a system that addresses both points of view, with the goal to provide some stickiness of scientific open sharing, but also the possibility to operate with scientific material that require restrictiveness, for example due to privacy rights.
We suggest the following clause for the reuse of open research data:
Upon publication of any scientific work under a broad definition (including, but not limited to journal papers, books or book chapters, conference proceedings, blog posts) that is based in full or in part on this data set, all data analysis scripts involved in the creation of this work must be made openly available under a license that allows reuse (e.g., BSD or MIT).
(Of course more topics must be addressed in the license, such as the obligation to properly cite the authors of the data set, not to try to reidentify research participants, etc. But we focus only on the copyleft aspect here).
This system has some differences from traditional copyleft licenses.
The proposed system offers some protection against the “research parasites” argument. The parasite discussion refers to the free-rider problem in social dilemmas: While some people invest resources to provide a public good, others (the parasites/free-riders) profit from the public good, without giving back to the community (see also Linek et al., 2017). This often creates a feeling of injustice, and impulses to punish the free-riders. (An entire scientific field is devoted to the structural, sociological, political, and psychological properties and consequences of such social dilemma structures.)
In the proposed licensing system, those who profit from openness by reusing open data must give something back to the community. This increases overall openness, reusability, and reproducibility of scientific outputs, and probably decreases feelings of exploitation and unfairness for the data providers.
Do you think such a license would work? Do you see any drawbacks we didn’t think of?
You can leave feedback here as a comment, on Twitter (@nicebread303) or via email to felix@nicebread.de.
Researchers of motivational psychology have long struggled with the power motive’s heterogeneous definition encompassing elements such as desires for dominance, reputation, prestige, leadership, and status (e.g., Winter, 1973). This heterogeneity has likely been responsible for researchers having found different relationships between the power motive and external variables depending on which power motive scale they used (e.g., Engeser & Langens, 2010). Thus, to provide a long-needed taxonomy of clearly distinguishable power motive components we developed the dominance, prestige, and leadership (DoPL) account of social power motives. In particular we differentiate between:
The dominance motive, defined as a desire for coercive power obtained through threats, intimidation, or deception
The prestige motive, defined as a desire for voluntary deference obtained through others’ admiration and respect particularly for one’s valued skills and knowledge
The leadership motive, defined as a desire for legitimised power granted by one’s group and obtained through taking responsibility in and for this group
Opposed to previous attempts to differentiate different power motive components (e.g., socialised and personalised power; McClelland, 1970) the DoPL account of social power motives is based on a solid theoretical framework adapted from research into social hierarchies (e.g., Cheng, Tracy, & Henrich, 2010; Henrich & Gil-White, 2001). Thus, the DoPL account does not suffer from strongly different interpretations of how these components manifest themselves.
Using newly developed DoPL questionnaires we showed the DoPL motives can be measured both reliably and distinctively (study 1). Moreover, we showed these DoPL motives strongly related to a common power desire (study 2), explaining more than 80% of variance in two established power motive scales (UMS power, Schönbrodt & Gerstenberg, 2012; PRF dominance, Jackson, 1984). Assessing their nomological networks (studies 3 & 4), we demonstrated distinct associations such as between…
the dominance motive and self-reported anger and verbal aggression
the prestige motive and self-reported fear of losing reputation and claiming to have higher moral concerns
the leadership motive and self-reported emotional stability and helping behaviour
Regarding observed behaviour and other external variables (studies 5 to 7) we found:
The dominance motive uniquely and negatively predicted the amount of money given to another player in a dictator game after having received nothing in two previous dictator games. This effect can be explained by a combination of general agonistic tendencies as well as retaliatory desires related to the dominance motive.
The leadership motive uniquely predicted the attainment of higher employment ranks across all kinds of professions. This effect was somewhat stronger in females which might be explained by discrimination against females regarding promotions and thus females having to compensate by being more highly motivated to reach high leadership positions.
When donating behaviour to a charity was made overt, residualised dominance motives (i.e., controlled for shared prestige and leadership influences) related negatively to the overall proportion donated to a charity as well as the probability to donate. Whereas residualised leadership motives only related positively to the overall amount donated to charity, residualised prestige motives only related positively to the probability to donate. Thus, to some degree, dominance desires relate negatively and leadership and prestige desires positively to prosocial donating behaviour.
This research shows that different power motive components in many (but not all) cases relate differently to a range of external variables. Thus, to improve the prediction of influential power-relevant behaviour as a function of individuals’ power desires we invite researchers to employ this novel taxonomy of power motives to further advance this important field of research.
Cheng, J. T., Tracy, J. L., & Henrich, J. (2010). Pride, personality, and the evolutionary foundations of human social status. Evolution and Human Behavior, 31, 334–347 https://doi.org/10.1016/
Engeser, S., & Langens, T. (2010). Mapping explicit social motives of achievement, power, and affiliation onto the five-factor model of personality. Scandinavian Journal of Psychology, 51, 309–318 https://doi.org/10.1111/j.1467-9450.2009.00773.x.
Henrich, J., & Gil-White, F. J. (2001). The evolution of prestige: Freely conferred deference as a mechanism for enhancing the benefits of cultural transmission. Evolution and Human Behavior, 22, 165–196 https://doi.org/10.1016/S1090-5138(00)00071-4.
Jackson, D. N. (1984). Personality research form manual (3rd ed.). Port Huron: Research Psychologists Press.
McClelland, D. C. (1970). The two faces of power. Journal of International Affairs, 24, 29–47.
Schönbrodt, F. D., & Gerstenberg, F. X. R. (2012). An IRT analysis of motive questionnaires: The unified motive scales. Journal of Research in Personality, 46, 725–742 https://doi.org/10.1016/j.jrp.2012.08.010. [Free PDF on OSF]
Winter, D. G. (1973). The power motive. New York: The Free Press.
]]>Let us start with the HARKer: Since the conducted hypothesis tests in our defined scenario are essentially independent, the situation can be seen as a problem of multiple testing. This means, it is comparatively easy to determine the exact probability that the HARKer will end up with at least one false-positive result given a certain number of hypothesis tests. Assuming no effects in the population (for example, no correlation between the variables), one can picture the situation as a decision tree: At each branch level stands a hypothesis test which can either result in a non-significant result with 95% probability or in a (spurious) significant result with 5% probability, which is the level.
No matter how many hypothesis tests the HARKer conducts, there will only be one condition in the all-null scenario where no error occurs, that is, where all hypothesis tests yield non-significant results. The probability that this occurs can be calculated by , with being the number of conducted hypothesis tests. The probability that at least one of the hypothesis tests is significant is the probability of the complementary event, that is . For example, when the HARKer computes hypothesis with an alpha level of , the overall probability to obtain at least one false positive result is . Of course, the formula can be adjusted for other suggested alpha levels, such as or . We show this general formula in the R-code chunk below.
The Accumulator has a different tactic: Instead of conducting multiple hypothesis tests on different variables of one data set, he repeatedly conducts the same hypothesis test on the same variables in a growing sample. Starting with a minimum sample size, the Accumulator looks at the results of the analysis – if these are significant, data collection is stopped, if not, more data is collected until (finally) the results become significant, or a maximum sample size is reached. This strategy is also called Optional Stopping. Of course, the more often a researcher peeks at the data, the higher is the probability to obtain a false positive result at least once. However, this overall probability is not the same as the one obtained through HARKing. The reason is that the hypothesis tests are not independent in this case. Why is that? The same hypothesis test is repeatedly conducted on only slightly different data. In fact, the data that were used in in the first hypothesis test are used in every single of the subsequent hypothesis tests so that there is a spillover effect of the first test to every other hypothesis test in the set. Imagine, your initial sample contains an outlier: This outlier will affect the test results in any other test. With multiple testing, in contrast, the outlier will affect only the test in question but none of the other tests in the set.
So does this dependency make optional stopping more or less effective than HARKing? Of course, people have been wondering about this for quite a while. A paper by Armitage et al. (1969) demonstrates error accumulation in optional stopping for three different tests. We can replicate their results for the z-test with a small simulation (a more flexible simulation can be found at the end of the blog post): We start by drawing a large number of samples (iter) with the maximum sample size (n.max) from the null hypothesis. Then we conduct a sequential testing procedure on each of the samples, starting with a minimum sample size (n.min) and going up in several steps (step) up to the maximum sample size. The probability to obtain a significant result at least once up to a certain step can be estimated through the percentage of iterations that end up with at least one significant result at that point.
For example, the researcher conducts a two-sided one-sample z-test with an overall level of .05 in a sequential way. He starts with 10 observations, then always adds another 10 if the result is not significant, up to 100 observations at maximum. This means, he has 10 chances to peek at the data and end the data collection if the hypothesis test is significant. Using our simulation function, we can determine the probability to have obtained at least one false positive result at any of these steps:
We can see that with one single evaluation, the false positive rate is at the nominal 5%. However, when more in-between tests are calculated, the false positive rate rises to roughly 20% with ten peeks. This means that even if there is no effect at all in the population, the researcher would have stopped data collection with a signficant result in 20% of the cases.
Let’s compare the false positive rates of HARKing and optional stopping: Since the researcher in our example above conducts one to ten dependent hypothesis tests, we can compare this to a situation where a HARKer conducts one to ten independent hypothesis tests. The figure below shows the results of both p-hacking strategies:
We can see that HARKing produces higher false positive rates than optional stopping with the same number of tests. This can be explained through the dependency on the first sample in the case of optional stopping: Given that the null hypothesis is true, this sample is not very likely to show extreme effects in any direction (however, there is a small probability that it does). Every extension of this sample has to “overcome” this property not only by being extreme in itself but also by being extreme enough to shift the test on the overall sample from non-significance to significance. In contrast, every sample in the multiple testing case only needs to be extreme in itself. Note, however, that false positive rates in optional stopping are not only dependent on the number of interim peeks, but also on the size of the initial sample and on the step size (how many observations are added between two peeks?). The difference between multiple testing and optional stopping which you see in the figure above is therefore only valid for this specific case. Going back to the two researchers from our example, we can say that the HARKer has a better chance to come up with significant results than the Accumulator, if both do the same number of hypothesis tests.
You can use the interactive p-hacker app to experience the efficiency of both p-hacking strategies yourself: You can increase the number of dependent variables and see whether one of them gets significant (HARing), or you can got to the “Now: p-hack!” tab and increase your sample until you obtain significance. Note that the DVs in the p-hacker app are not completely independent as in our example above, but rather correlate with r = .2, assuming that the DVs to some extent measure at least related constructs.
To conclude, we have shown how two p-hacking techniques work and why their application is bad for science. We found out that p-hacking techniques based on multiple testing typically end up with higher rates of false positive results than p-hacking techniques based on optional stopping, if we assume the same number of hypothesis tests. We want to stress that this does not mean that naive optional stopping is okay (or even okay-ish) in frequentist statistics, even if it does have a certain appeal. For those who want to do guilt-free optional stopping, there are ways to control for the error accumulation in the frequentist framework (see for example Wald, 1945, Chow & Chang, 2008, Lakens, 2014) and sequential Bayesian hypothesis tests (see for example our paper on sequential hypothesis testing with Bayes factors or Rouder, 2014).
Our department embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open materials, and study pre-registration. Candidates are asked to describe in what way they already pursued and plan to pursue these goals.
Since then, every professorship announcement contained this paragraph (and we made good experiences with it).
I am very happy to announce that my department now turned this implicit policy into an explicit hiring policy, effective since May 2018: The department’s steering committee unanimously voted for an explicit policy to always include this (or a similar) statement to all future professorship job advertisements.
It is the task of the appointment committee to value the existing open science activities as well as future commitments of applicants appropriately. By including this statement, our department aims to communicate core values of good scientific practice and to attract excellent researchers who aim for transparent and credible research.
In this respect, take a look at the current draft of a Modular Certification Initiative (initiated by Chris Chambers, Kyle MacDonald and me, with a lot of input from the open science community). With this TOP-like scheme, institutions, but also single researchers, can select a level of openness which they require in their hiring process.
So, if you want to join the LMU psychology department as a professor, you should better have some open science track record.
Previous investigations typically looked only at publication bias or questionable research practices QRPs (but not both), used non-representative study-level sample sizes, or only compared few bias-correcting techniques, but not all of them. Our goal was to simulate a research literature that is as realistic as possible for psychology. In order to simulate several research environments, we fully crossed five experimental factors: (1) the true underlying effect, δ (0, 0.2, 0.5, 0.8); (2) between-study heterogeneity, τ (0, 0.2, 0.4); (3) the number of studies in the meta-analytic sample, k (10, 30, 60, 100); (4) the percentage of studies in the meta-analytic sample produced under publication bias (0%, 60%, 90%); and (5) the use of QRPs in the literature that produced the meta-analytic sample (none, medium, high).
This blog post summarizes some insights from our study, internally called “meta-showdown”. Check out the preprint; and the interactive app metaExplorer. The fully reproducible and reusable simulation code is on Github, and more information is on OSF.
In this blog post, I will highlight some lessons that we learned during the project, primarily focusing on what not do to when performing a meta-analysis.
Constraints on Generality disclaimer: These recommendations apply to typical sample sizes, effect sizes, and heterogeneities in psychology; other research literatures might have different settings and therefore a different performance of the methods. Furthermore, the recommendations rely on the modeling assumptions of our simulation. We went a long way to make them as realistic as possible, but other assumptions could lead to other results.
If studies have no publication bias, nothing can beat plain old random effects meta-analysis: it has the highest power, the least bias, and the highest efficiency compared to all other methods. Even in the presence of some (though not extreme) QRPs, naive RE performs better than all other methods. When can we expect no publication bias? If (and, in my opinion only if) we meta-analyze a set of registered reports.
But.
In any other setting except registered reports, a consequential amount of publication bias must be expected. In the field of psychology/psychiatry, more than 90% of all published hypothesis tests are significant (Fanelli, 2011) despite the average power being estimated as around 35% (Bakker, van Dijk, & Wicherts, 2012) – the gap points towards a huge publication bias. In the presence of publication bias, naive random effects meta-analysis and trim-and-fill have false positive rates approaching 100%:
More thoughts about trim-and-fill’s inability to recover δ=0 are in Joe Hilgard’s blog post. (Note: this insight is not really new and has been shown multiple times before, for example by Moreno et al., 2009, and Simonsohn, Nelson, and Simmons, 2014).
Our recommendation: Never trust meta-analyses based on naive random effects and trim-and-fill, unless you can rule out publication bias. Results from previously published meta-analyses based on these methods should be treated with a lot of skepticism.
Update 2017/06/09: We had a productive exchange with Uri Simonsohn and Joe Simmons concerning what should be estimated in a meta-analysis with heterogeneity. Traditionally, meta-analysts have tried to arrive at techniques that recover the true average effect of all conducted studies (AEA – average effect of all studies). Simonsohn et al (2014) propose estimating a different magnitude; the average effect of the studies one observes, rather than of all studies (AEO – average effect of observed studies). See Simonsohn et al (2014), the associated Supplementary Material 2, and also this blog post for arguments why they think this is a more useful quantity to estimate.
Hence, an investigation of the topic can be done on two levels: A) What is the more appropriate estimand (AEA or AEO?), and B) Under what conditions are estimators able to recover the respective true value with the least bias and least variance?
Instead of updating the section of the current blog post in the light of our discussion, I decided to cut it out and to move the topic to a future blog post. Likewise, one part of our manuscript’s revision will be a more detailed discussion about excatly these differences.
I archived the previous version of the blog post here.
Many bias-correcting methods are driven by QRPs – the more QRPs, the stronger the downward correction. However, this effect can get so strong, that methods overadjust into the opposite direction, even if all studies in the meta-analysis are of the same sign:
Note: You need to set the option “Keep negative estimates” to get this plot.
Our recommendation: Ignore bias-corrected results that go into the opposite direction; set the estimate to zero, do not reject H₀.
Typical small-study effects (e.g., by p-hacking or publication bias) induce a negative correlation between sample size and effect size – the smaller the sample, the larger the observed effect size. PET-PEESE aims to correct for that relationship. In the absence of bias and QRPs, however, random fluctuations can lead to a positive correlation between sample size and effect size, which leads to a PET and PEESE slope of the unintended sign. Without publication bias, this reversal of the slope actually happens quite often.
See for example the next figure. The true effect size is zero (red dot), naive random effects meta-analysis slightly overestimates the true effect (see black dotted triangle), but PET and PEESE massively overadjust towards more positive effects:
As far as I know, PET-PEESE is typically not intended to correct in the reverse direction. An underlying biasing process would have to systematically remove small studies that show a significant result with larger effect sizes, and keep small studies with non-significant results. In the current incentive structure of psychological research, I see no reason for such a process, unless researchers are motivated to show that a (maybe politically controversial) effect does not exist.
Our recommendation: Ignore the PET-PEESE correction if it has the wrong sign, unless there are good reasons for an untypical selection process.
A bias can be more easily accepted if it always is conservative – then one could conclude: “This method might miss some true effects, but if it indicates an effect, we can be quite confident that it really exists”. Depending on the conditions (i.e., how much publication bias, how much QRPs, etc.), however, PET/PEESE sometimes shows huge overestimation and sometimes huge underestimation.
For example, with no publication bias, some heterogeneity (τ=0.2), and severe QRPs, PET/PEESE underestimates the true effect of δ = 0.5:
In contrast, if no effect exists in reality, but strong publication bias, large heterogeneity and no QRPs, PET/PEESE overestimates at lot:
In fact, the distribution of PET/PEESE estimates looks virtually identical for these two examples, although the underlying true effect is δ = 0.5 in the upper plot and δ = 0 in the lower plot. Furthermore, note the huge spread of PET/PEESE estimates (the error bars visualize the 95% quantiles of all simulated replications): Any single PET/PEESE estimate can be very far off.
Our recommendation: As one cannot know the condition of reality, it is probably safest not to use PET/PEESE at all.
Again, please consider the “Constraints on Generality” disclaimer above.
As with any general recommendations, there might be good reasons to ignore them.
Now we can compare two completely independently coded p-curve disclosure tables about a large set of articles. Any disagreement of course does not mean that one party is right and the other is wrong. But it will be interesting to see the amount of agreement.
Here comes Anna’s blog post about her own study. Anna Bittner is now doing her Master of Finance at the University of Melbourne.
by Anna Bittner and Felix Schönbrodt
The recent discoveries on staggeringly low replicability in psychology have come as a shock to many and led to a discussion on how to ensure better research practices are employed in the future. To this end, it is necessary to find ways to efficiently distinguish good research from bad and research that contains evidential value from such that does not.
In the past the impact factor (IF) has often been the favored indicator of a journal’s quality. To check whether a journal with a higher IF does indeed publish the “better” research in terms of evidential value, we compared two academic journals from the domain of social psychology: The Journal of Personality and Social Psychology (JPSP, Impact Factor = 5.031) and the Journal of Applied Social Psychology (JASP, Impact Factor = 0.79).
For this comparison, Anna has analysed and carefully hand-coded all studies with hypothesis tests starting in January 2013 and progressing chronologically until about 110 independent test statistics for each journal were acquired. See the full report (in German) in Anna’s bachelor thesis. These test statistics were fed into the p-checker app (Version 0.4; Schönbrodt, 2015) that analyzed them with the tools p-curve, TIVA und R-Index.
All material and raw data is available on OSF: https://osf.io/pgc86/
P-curve (Simonsohn, Nelson, & Simmons, 2014) takes a closer look at all significant p-values and plots them against their relative frequency. This results in a curve that will ideally have a lot of very small p-values (<.025) and much fewer larger p–values (>.025). Another possible shape is a flat curve, which will occur when researchers only investigate null effects and selectively publish those studies that obtained a p-value < .05 by chance. under the null hypothesis each individual p-value is equally likely and the distribution is even. P-curve allows to test whether the empirical curve is flatter than the p-curve that would be expected at any chosen power.
Please note that p-curve assumes homogeneity (McShane, Böckenholt, & Hansen, 2016). Lumping together diverse studies from a whole journal, in contrast, guarantees heterogeneity. Hence, trying to recover the true underlying effect size/power is of limited usefulness here.
The p-curves of both JPSP and JASP were significantly right-skewed, which suggests that both journal’s output cannot be explained by pure selective publication of null effects that got significant by chance. JASP’s curve, however, had a much stronger right-skew, indicating stronger evidential value:
TIVA (Schimmack, 2014a) tests for an insufficient variance of p-values: If no p-hacking and no publication bias was present, the variance of p-values should be at least 1. An value below 1 is seen as indicator of publication bias and/or p-hacking. However, variance can and will be much larger than 1 when studies of different sample and effect sizes are included in the analysis (which was inevitably the case here). Hence, TIVA is a rather weak test of publication bias when heterogeneous studies are combined: A p-hacked set of heterogeneous effect sizes can still result in a high variance in TIVA. Publication bias and p-hacking reduce the variance, but heterogeneity and different sample sizes can increase the variance in a way that the TIVA is clearly above 1, even if all studies in the set are severely p-hacked.
As expected, neither JPSP nor JASP attained a significant TIVA result, which means the variance of p-values was not significantly smaller than 1 for either journal. Descriptively, JASP had a higher variance of 6.03 (chi2(112)=674, p=1), compared to 1.09 (chi2(111)=121, p=.759) for the JPSP. Given the huge heterogeneity of the underlying studies, a TIVA variance in JPSP close to 1 signals a strong bias. This, again, is not very surprising. We already knew before with certainty that our literature is plagued by huge publication bias.
The descriptive difference in TIVA variances can be due to less p-hacking, less publication bias, or more heterogeneity of effect sizes and sample sizes in JASP compared to JPSP. Hence, drawing firm conclusions from this numerical difference is difficult; but the much larger value in JASP can be seen as an indicator that the studies published there paint a more realistic picture.
(Note: The results here differ from the results reported in Anna’s bachelor thesis, as the underlying computation has been improved. p-checker now uses logged p-values, which allows more precision with very small p-values. Early versions of p-checker underestimated the variance when extremely low p-values were present).
Unfortunately, Motyl et al. do not report the actual variances from their TIVA test (only the test statistics), so a direct comparison of our results is not possible.
The R-Index (Schimmack, 2014b) is a tool that aims to quantify the replicability of a set of studies. It calculates the difference between median estimated power and success rate, which results in the so called inflation rate. This inflation rate is then subtracted from the median estimated power, resulting in the R-Index. Here is Uli Schimmack’s interpretation of certain R-Index values: “The R-Index is not a replicability estimate […] I consider an R-Index below 50 an F (fail). An R-Index in the 50s is a D, and an R-Index in the 60s is a C. An R-Index greater than 80 is considered an A”.
Here, again, the JASP was ahead. It obtained an R-Index of .60, whereas the JPSP landed at .49.
Both journals had success rates of around 80%, which is much higher than what would be expected with the average power and effect sizes found in psychology (Bakker, van Dijk, & Wicherts, 2012). It is known and widely accepted that journals tend to publish significant results over non-significant ones.
Motyl et al. report an R-Index of .52 for 2013-2014 for high impact journals, which is very close to our value of .49.
The comparison between JPSP and JASP revealed a better R-Index, a more realistic TIVA variance and a more right-skewed p-curve for the journal with the much lower IF. As the studies had roughly comparable sample sizes (JPSP: Md = 86, IQR: 54 – 124; JASP: Md = 114, IQR: 65 – 184), I would bet some money that more studies from JASP replicate then from JPSP.
A journal’s prestige does not protect it from research submissions that contain QRPs – contrarily it might lead to higher competition between reseachers and more pressure to submit a significant result by all means. Furthermore, higher rejection rates of a journal also leave more room for “selecting for significance”. In contrast, a journal that must publish more or less every submission it gets to fill up its issues simply does not have much room for this filter. With the currently applied tools, however, it is not possible to make a distinction between p-hacking and publication bias: they only detect patterns in test statistics that can be the result of both practices.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543-554.
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016). Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11, 730–749. doi:10.1177/1745691616662243
Schimmack, U. (2014b). Quantifying Statistical Research Integrity: The Replicabilty-Index.
Schimmack, U. (2014a, December 30). The Test of Insufficient Variance (TIVA): A New Tool for the Detection of Questionable Research Practices. Retrieved from https://replicationindex.wordpress.com/2014/12/30/the-test-of-insufficient-variance-tiva-a-new-tool-for-the-detection-of-questionable-research-practices/
Schimmack, U. (2015, September 15). Replicability-Ranking of 100 Social Psychology Departments [Web log post]. Retrieved from https://replicationindex.wordpress.com/2015/09/15/replicability-ranking-of-100-social-psychology-departments/
Schönbrodt, F. (2015). p-checker [Online application]. Retrieved from http://shinyapps.org/apps/p-checker/
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534.
In the last year, the discussion in our field moved from “Do we have a replication crisis?” towards “Yes, we have a problem, and what can and should we change? How can be implement it?”. I think that we need both top-down changes on an institutional level, combined with bottom-up approaches, such as local Open Science Initiatives. Here, I want to present one big institutional change concerning open data.
The German Research Foundation (DFG), the largest public funder of research in Germany, updated their policy on data sharing, which can be summarized in a single sentence: Publicly funded research, including the raw data, belongs to the public. Consequently, all research data from a DFG funded project should be made open immediately, or at least a couple of months after finalization of the research project (see [1] and [2]). Furthermore, the DFG asked all scientific disciplines to develop more specific guidelines which implement these principles in their respective discipline.
The German Psychological Society (Deutsche Gesellschaft für Psychologie, DGPs) installed a working group (Andrea Abele-Brehm, Mario Gollwitzer and me) who worked for one year on such recommendations for psychology.
In the development of the document, we tried to be very inclusive and to harvest the wisdom of the crowd. A first draft (Feb 2016) was discussed for 6 weeks in an internet forum where all DGPs members could comment. Based on this discussion (and many additional personal conversations), a revised version was circulated and discussed in person with a smaller group of interested members (July 2016) and a representative of the DFG. Furthermore, we had regular contact to the “Fachkollegium Psychologie” of the DFG (i.e., the group of people which decides about funding decisions in psychology; meanwhile, the members of the Fachkollegium have changed on a rotational basis). Finally, the chair persons of all sections of the DGPs and the speakers of the young members had another possibility to comment. On September 17, the recommendations were officially adopted by the society.
I think this thorough and iterative process was very important for two reasons: First, it definitely improved the quality of the document, because we got so many great ideas and comments from the members, ironing out some inconsistencies and covering some edge cases. Second, it was important in order to get people on board. As this new open data guideline of the DFG causes a major change in the way we do our everyday scientific work, we wanted to talk to and convince as many people as possible from the early steps on. Of course not every single of the >4,000 members is equally convinced, but the topic now has considerable attention in the society.
Hence, one focus was consensus and inclusivity. At the same time, we had the goal to develop bold and forward-looking guidelines that really address the current challenges of the field, and not to settle on the lowest common denominator. For this goal, we had to find a balance between several, sometimes conflicting, values.
Research transparency ⬌ privacy rights. A first specialty of psychology is that we do not investigate rocks or electrons, but human subjects who have privacy rights. In a nutshell, privacy rights have to be respected, and in case of doubt they win over openness. But if data can be properly anonymized, there’s no problem in open sharing; one possibility to share non-anonymous research data are “scientific use files”, where access is restricted to scientists. If data cannot be shared due to privacy (or other) reasons, this has to be made transparent in the paper. (Hence, the recommendations are PRO compatible). The recommendations give clear guidance on privacy issues and gives practical advice, for example, on how to write your informed consent that you actually are able to share the data afterwards.
Data reuse ⬌ right of first usage. A second balance concerns an optimal reuse of data on the one hand, and the right of first usage of the original authors. In the discussion phase during the development of the recommendations, several people expressed the fear of “research parasites”, who “steal” the data from hard-working scientists. A very common gut feeling is: “The data belong to me”. But, as we are publicly funded researchers with publicly funded research projects, the answer is quite clear: the data belong to the public. There is no copyright on raw data. On the other hand, we also need incentives for original researchers to generate data in the first place. Data generators of course have the right of first usage, and the recommendations allow to extend this right by an embargo of 5 more years (see below). But at the end of the day, publicly funded research data belongs to the public, and everybody can reuse it. If data are open by default, a guideline also must discuss and define how data reuse should be handled. Our recommendations make suggestions in which cases a co-authorship should be offered to the data providers and in which cases this is not necessary.
Verification ⬌ fair treatment of original authors. Finally, research should be verifiable, but with a fair treatment of the original authors. The guidelines say that whenever a reanalysis of a data set is going to be published (and that also includes blog posts or presentations), the original authors have to be informed about this. They cannot prevent the reanalysis, but they have the chance to react to it.
We distinguish two types of data sharing:
Type 1 data sharing means that all raw data should be openly shared that is necessary to reproduce the results reported in a paper. Hence, this can be only a subset of all available variables in the full data set: The subset which is needed to reproduce these specific results. The primary data are an essential part of an empirical publication, and a paper without that simply is not complete.
Type 2 data sharing refers to the release of the full data set of a funded research project. The DGPs recommendations claim that after the end of a DFG-funded
project all data – even data which has not yet been used for publications – should be made open. Unpublished null results, or additional, exploratory variables now have to chance to see the light and to be reused by other researchers. Experience tells that not all planned papers have been written after the official end date of a project. Therefore, the recommendations allow that the right of first usage can be extended with an embargo period of up to 5 years, where the (so far unpublished) data do not have to be made public. The embargo option only applies to data that has not yet been used for publications. Hence, typically an embargo cannot be applied to Type 1 data sharing.
To summarize, I think these recommendations are the most complete, practical, and specific guidelines for data sharing in psychology to date. (Of course much more details are in the recommendations themselves). They fully embrace openness, transparency and scientific integrity. Furthermore, they do not proclaim detached ethical principles, but give very practical guidance on how to actually implement data sharing in psychology.
What are the next steps? The president of the DGPs, Prof. Conny Antoni, and the secretary Prof. Mario Gollwitzer already contacted other psychological societies (APA, APS, EAPP, EASP, EFPA, SIPS, SESP, SPSP) and introduced our recommendations. The Board of Scientific Affairs of EFPA – the European Federation of Psychologists’ Associations – already expressed its appreciation of the recommendations and will post them on their website. Furthermore, it will discuss them in an invited symposium on the European Congress of Psychology in Amsterdam this year. A mid-term goal will also be to check compatibility with existing other guidelines and to think about a harmonization of several guidelines within psychology.
As other scientific disciplines in Germany also work on their specific implementations of the DFG guidelines, it will be interesting to see whether there are common lines (although there certainly will be persisting and necessary differences between the requirements of the fields). Finally, we are in contact with the new Fachkollegium at the DFG, with the goal to see how the recommendations can and should be used in the process of funding decisions.
If your field also implements such recommendations/guidelines, don’t hesitate to contact us.
Schönbrodt, F., Gollwitzer, M., & Abele-Brehm, A. (2017). Der Umgang mit Forschungsdaten im Fach Psychologie: Konkretisierung der DFG-Leitlinien. Psychologische Rundschau, 68, 20–35. doi:10.1026/0033-3042/a000341. [PDF German][PDF English]
(English translation by Malte Elson, Johannes Breuer, and Zoe Magraw-Mickelson)
This is the second part of “Two meanings of priors”. The first part explained a first meaning – “priors as subjective probabilities of models”. While the first meaning of priors refers to a global appraisal of existing hypotheses, the second meaning of priors refers to specific assumptions which are needed in the process of hypothesis building. The two kinds of priors have in common that they are both specified before concrete data are available. However, as it will hopefully become evident from the following blog post, they differ significantly from each other and should be distinguished clearly during data analysis.
In order to know how well evidence supports a hypothesis compared to another hypothesis, one must know the concrete specifications of each hypothesis. For example, in the tea tasting experiment, each hypothesis was characterized by a specific probability (e.g., the success rate of exactly 0.5 in H_{Fisher }of the previous blog post). What might sound trivial at first – deciding on the concrete specifications of a hypothesis – is in fact one of the major challenges when doing Bayesian statistics. Scientific theories are often imprecise, resulting in more than one plausible way to derive a hypothesis. With deciding upon one specific hypothesis, often new auxiliary assumptions are made. These assumptions, which are needed in order to specify a hypothesis adequately, are called “priors” as well. They influence the formulation and interpretation of the likelihood (which gives you the plausibility of data under a specific hypothesis). We will illustrate this in an example.
A food company conducts market research in a large German city. They know from a recent representative enquiry by the German Federal Statistical Office that Germans spend on average 4.50 € for their lunch (standard deviation: 0.60 €). Now they want to know if the inhabitants of one specific city spend more money for their lunch compared to the German average. They expect lunch expenses to be especially high in this city because of the generally high living costs. In a traditional testing procedure in inferential statistics the food company would formulate two hypotheses to test their assumption: a null and an alternative hypothesis: H_{0}: µ ≤ 4.50 and H_{1}: µ > 4.50.
In Bayesian hypothesis testing, the formulation of the hypotheses has to be more precise than this. We need precise hypotheses as a basis for the likelihood functions which assign probability values to possible states of reality. The traditional formulation, µ > 4.50, is too vague for that purpose: Is any lunch cost above 4.50€ a priori equally likely? Is it plausible that a lunch costs 1,000,000€ on average? Probably not. Not every state of reality is, a priori, equally plausible. “Models connect theory to data“ (Rouder, Morey, & Wagenmakers, 2016), and a model that predicts everything predicts nothing.
As Bayesian statisticians we therefore must ask ourselves: Which values are more plausible given that our hypotheses are true? Of course, our knowledge differs from case to case in this point. Sometimes, we may be able to commit to a very small range of plausible values or even to a single value (in this case, we would call the respecting hypothesis a “point hypothesis”). Theories in physics sometimes predict a single state of reality: “If this theory is true, then the mass of a Chicks boson is exactly 1.52324E-16 gr”.
More often, however, our knowledge about plausible values under a certain theory might be less precise, leading to a wider range of plausible values. Hence, the prior in the second sense defines the probability of a parameter value given a hypothesis, p(θ | H1).
Let us come back to the food company example. Their null hypothesis might be that there is no difference between the city in the focus of their research project and the German average. Hence, the null hypothesis predicts an average lunch cost of 4.50€. With the alternative hypothesis, it becomes slightly more complex. They assume that average lunch expenses in the city should be higher than the German average, so the most plausible value under the alternative hypothesis should be higher than 4.5. However, they may deem it very improbable that the mean lunch expenses are more than two standard deviations higher than the German average (so, for example, it should be very improbable that someone spends more than, say, 10 EUR for lunch even in the expensive city). With this knowledge, they can put most plausibility on values in a range from 4.5 to 5.7 (4.5 + 2 standard deviations). They could further specify their hypothesis by claiming that the most plausible value should be 5.1, i.e., one standard deviation higher than the German average. The elements of these verbal descriptions of the alternative hypothesis can be summarized in a truncated normal distribution that is centered over 5.1 and truncated at 4.5 (as the directional hypothesis does not predict values in the opposite direction).
With this model specification, the researchers would place 13% of the probability mass on values larger than 2SD of the general population (i.e., > 5.7).
Making it even more complex, they could quantify their uncertainty about the most plausible value (i.e., the maximum of the density distribution) by assigning another distribution to it. For example, they could build a normal distribution around it, with a mean of 5.1 and a standard deviation of 0.3. This would imply that in their opinion, 5.1 is the “most plausible most plausible value” but that values between 4.8 and 5.4 are also potential candidates for the most plausible value.
What you can notice in the example about the development of hypotheses is that the market researchers have to make auxiliary assumptions on top of their original hypothesis (which was H1: µ > 4.5). If possible, these prior plausibilities should be informed by theory or by previous empirical data. Specifying alternative hypothesis in this way may seem to be an unnecessary burden compared to traditional hypothesis testing where these extra assumptions seemingly are not necessary. Except that they are necessary. Without going into detail in this blog post, we recommend to read Rouder et al.’s (2016a) “Is there a free lunch in inference?“, with the bottom line that principled and rational inference needs specified alternative hypotheses. (For example, in Neyman-Pearson testing, you also need to specify a precise alternative hypothesis that refers to the “smallest effect size of interest”)
Furthermore, readers might object: “Researchers rarely have enough background knowledge to specify models that predict data“. Rouder et al. (2016b) argue that this critique is overstated, as (1) with proper elicitation, researchers often know much more than they initially think, (2) default models can be a starting point if really no information is available, and (3) several models can be explored without penalty.
A question that may come to your mind soon after you understood the difference between the two kinds of priors is: If they both are called “priors”, do they depend on each other in some way? Does the formulation of your “personal prior plausibility of a hypothesis” (like the skeptical observer’s prior on Mrs. Bristol’s tea tasting abilities) influence the specification of your model (like the hypothesis specification in the second example) or vice versa?
The straightforward answer to this question is “no, they don’t”. This can be easily illustrated in a case where the prior conviction of a researcher runs against the hypothesis he or she wants to test. The food company in the second example has sophisticatedly determined the likelihood of the two hypotheses (H0 and H1), which they want to pit against each other. They are probably considerably convinced that the specification of the alternative hypothesis describes reality better than the specification of the null hypothesis. In a simplified form, their prior odds (i.e., priors in the first sense) can be described as a ratio like 10:1. This would mean that they deem the alternative hypothesis ten times as likely as the null hypothesis. However, another food company, may have prior odds of 3:5 while conducting the same test (i.e., using the same prior plausibilities of model parameters). This shows that priors in the first sense are independent of priors in the second sense. Priors in the first sense change with different personal convictions while priors in the second sense remain constant. Similarly, prior beliefs can change after seeing the data – the formulation of the model (i.e., what a theory predicts) stays the same. (As long as the theory, from which the model specification is derived, does not change. In an estimation context, the model parameters are updated by the data.)
The term “prior” has two meanings in the context of Bayesian hypothesis testing. The first one, usually applied in Bayes factor analysis, is equivalent to a prior subjective probability of a hypothesis (“how plausible do you deem a hypothesis compared to another hypothesis before seeing the data”). The second meaning refers to the assumptions made in the specification of the model of the hypotheses which are needed to derive the likelihood function. These two meanings of the term “prior” have to be distinguished clearly during data analysis, especially as they do not depend on each other in any way. Some researchers (e.g., Dienes, 2016) therefore suggest to call only priors in the first sense “priors” and speak about “specification of the model” when referring to the second meaning.
Read the first part of this blog post: Priors as the plausibility of models
Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on?. Perspectives On Psychological Science, 6(3), 274-290. http://doi:10.1177/1745691611406920
Dienes, Z. (2016). How Bayes factors change scientific practice. Journal Of Mathematical Psychology, 7278-89. http://doi:10.1016/j.jmp.2015.10.003
Lindley, D. V. (1993). The analysis of experimental data: The appreciation of tea and wine. Teaching Statistics, 15(1), 22-25. http://dx.doi.org/10.1111/j.1467-9639.1993.tb00252.x
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016a). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520–547. http://doi.org/10.1111/tops.12214
Rouder, J. N., Morey, R. D., & Wagenmakers, E. J. (2016b). The Interplay between Subjectivity, Statistical Practice, and Psychological Science. Collabra, 2(1), 6–12. http://doi.org/10.1525/collabra.28