Felix Schönbrodt

PD Dr. Dipl.-Psych.

In the era of #repligate: What are valid cues for the trustworthiness of a study?

[Update 2015/1/14: I consolidate feedback from Twitter, comments, email, and real life into the main text (StackExchange-style), so that we get a good and improving answer. Thanks to @TonyLFreitas, @PhDefunct, @bahniks, @JoeHilgard, @_r_c_a, @richardmorey, @R__INDEX, the commenters at the end of this post and on the OSF mailing list, and many others for their feedback!]

In a recent lecture I talked about the replication crisis in psychology. After the lecture my students asked: “We learn so much stuff in our lectures, and now you tell us that a considerable proportion of these ‘facts’ probably are just false positives, or highly exaggerated? Then, what can we believe at all?”. A short discussion soon led to the crucial question:

In the era of #repligate: What are valid cues for the trustworthiness of a study?
Of course the best way to judge a study’s quality would be to read the paper thoroughly, making an informed judgement about the internal and statistical validity, invest some extra time into a literature review, and maybe take a look at the raw data, if available. However, such an investment is not possible in all scenarios.

 

Here, I will only focus on cues that are easy and fast to retrieve.

 

As a conceptual framework we can use the lens model (Brunswick, 1956), which differentiates the concepts of cue usage and cue validity. We use some information as a manifest cue for a latent variable (“cue utilization”). But only some cues are also valid indicators (“cue validity”). Valid cues correlate with the latent variable, invalid cues have no correlation. Sometimes, there exist valid cues which we don’t use, and sometimes we use cues that are not valid. Of course, each of the following cues can be critiziced, and you certainly can give many examples where each cue breaks down. Furthermore, the absence of a positive cue (e.g., if a study has not been pre-registered, which was uncommon until recently) does not necessarily indicate the untrustworthiness.
But this is the nature of cues – they are not perfect, and only work on average.

 


Valid cues for trustworthiness of a single study:

  • Pre-registration. This might be one of the strongest cues for trustworthiness. Pre-registration makes p-hacking and HARKing unlikely (Wagenmakers, Wetzels, Borsboom, Maas, & Kievit, 2012), and takes care for a sufficient amount of statistical power (At least, some sort of sample size planning has been done. Of course, this depends on the correctness of the a-priori effect size estimate).
  • Sample size / Statistical Power. Larger samples mean higher power, higher precision, and less false positives (Bakker, van Dijk, & Wicherts, 2012; Maxwell, Kelley, & Rausch, 2008; Schönbrodt & Perugini, 2013). Of course sample size alone is not a panacea. As always, the garbage in/garbage out principle holds, and a well designed lab study with n=40 can be much more trustworthy than a sloppy mTurk study with n=800. But all other things being equal, I put more trust in larger studies.
  • Independent high-power replications. If a study has been independently replicated from another lab with high power and preferably pre-registered, this probably is the strongest evidence for the trustworthiness of a study (How to conduct a replication? See the Replication Recipe by Brandt et al., 2014).
  • I guess that studies with Open Data and Open Material have a higher replication rate
    • “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results” (Wicherts, Bakker, & Molenaar, 2011) —> this is not exactly Open Data, because here authors only shared data upon request (or not). But it points into the same direction.
    • Beyond publishing Open Data at all, the neatness of the data set and the quality of the analysis script is an indicator (see also comment by Richard Morey). The journal “Quarterly Journal of Political Science” demands to publish raw data and analysis code that generates all the results reported in the paper. Of these submissions, 54% “had results in the paper that differed from those generated by the author’s own code”! My fear is that analytical code that has not been refined and polished for publishing contains even more errors (not to speak of unreproducible point-and-click analyses). Therefore, a well prepared data set and analysis code should be a valid indicator.
    • Open Material could be an indicator that people are not afraid of replications and further scrutiny
  • An abstract with reasonable conclusions that stick close to the data – see also below: “Red flags”. This includes visible efforts of the authors to explain how they could be wrong and what precautions were/were not taken.
    • A sensitivity analysis, which shows that conclusions do not depend on specific analytical choices. For Bayesian analyses this means to explore how the conclusions depend on the choice of the prior. But you could also show how your results change when you do not exlude the outliers, or do not apply that debatable transformation to your data (see also comment)
  • Using the “21 Word Solution” of Simmons, Nelson, & Simonsohn (2012) leads to a better replication index.
These cues might be a feature of a specific study. Beyond that, these cues could also be used as indicators of an authors’ general approach to science (e.g., Does s/he in general embrace open practices and care about the replicability of his or her research? Does the author have a good replication record?). So the author’s open science reputation could be another valid indicator, and could be useful for hiring or tenure decisions.
(As a side note: I am not so interested in creating another formalized author index “The super-objective-h-index-extending-altmetric-open-science-author-index!”. But when I reflect about how I judge the trustworthiness of a study, I indeed take into account the open science reputation an author has).

Valid cues for trustworthiness of a research programme/ multiple studies:


Valid cues for UNtrustworthiness of a single study/ red flags:

In a comment below, Dr. R introduced the idea of “red flags”, which I really like. These red flags aren’t a prove of the untrustworthiness of a study – but definitely a warning sign to look closer and to be more sceptical.

  • Sweeping claims, counterintuitive, and shocking results (that don’t connect to the actual data)
  • Most p values are in the range of .03 – .05 (or, equivalently, most t-values in the 2-3 range, or most F-values are in the 4-9 range; see comment by Dr. R below).
    • How does a distribution of p values look like when there’s an effect? See DaniĂ«l Lakens blog. With large samples, p-values just below .05 even indicate support for the null!
  • It’s a highly cited result, but no direct replications have been published so far. That could be an indicator that many unsuccessful replication attempts went into the file-drawer (see comment by Ruben below).
  • Too good to be true: If several low-power studies are combined in a paper, it can be very unlikely that all of them produce significant results. The “Test of Excess Significance” has been used to formally test for “too many significant results”. Although this formal test has been criticized (e.g., see The Etz-Files, and especially the long thread of comments, or this special issue on the test), I still think excess significance can be used as a red flag indicator to look closer.

Possibly invalid cues (cues which are often used, but only seemingly are indicators for a study’s trustworthiness):

  • The journal’s impact factor. Impact factors correlate with retractions (Fang & Casadevall, 2011), but do not correlate with a single paper’s citation count (see here).
    • I’m not really sure whether that is a valid or invalid cue for a study’s quality. The higher retraction rate might due to the stronger public interest and a tougher post-publication review of papers in high-impact journals. The IF seems not to be predictive of a single paper’s citation count; but I’m not sure either whether the citation count is an index of a study’s quality. Furthermore, “Impact factors should have no place in grant-giving, tenure or appointment committees.” (ibid.), see also a reccent article by @deevybee in Times Higher Education.
    • On the other hand, the current replicability estimate of a full volume of JPSP is only at 20-30% (see Reproducibility Project: Psychology). A weak performance for one of our “best journals”.
  • The author’s publication record in high-impact journals or h-index. This might be a less valid cue as expected, or even an invalid cue.
  • Meta-analyses. Garbage-in, garbage-out: Meta-analyses of a biased literature produce biased results. Typical correction methods do not work well. When looking at meta-analyses, at least one has to check whether and how it was corrected for publication bias.
This list of cues was compiled in a collaborative effort. Some of them have empirical support; others are only a personal hunch.

 

So, if my students ask me again “What studies can we trust at all?”, I would say something like:
“If a study has a large sample size, Open Data, and maybe even has been pre-registered, I would put quite some trust into the results. If the study has been independently replicated, even better. In contrast to common practice, I do not care so much whether this paper has been published in a high-impact journal or whether the author has a long publication record. The next step, of course, is: Read the paper, and judge it’s validity and the quality of its arguments!”
What are your cues or tips for students?

This list certainly is not complete, and I would be interested in your ideas, additions, and links to relevant literature!

 

References

Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., et al. (2014). The Replication Recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. doi:10.1016/j.jesp.2013.10.005
Brunswik, E. (1956). Perception and the representative design of psychological experiments. University of California Press.
Fang, F. C., & Casadevall, A. (2011). Retracted science and the retraction index. Infection and Immunity, 79, 3855–3859. doi:10.1128/IAI.05661-11
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. doi:10.1146/annurev.psych.59.103006.093735
Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609–612. doi:10.1016/j.jrp.2013.05.009
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Maas, H. L. J. v. d., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
Comments (16) | Trackback

Gaze-cueing and trustworthiness: New paper + raw data + R script on OSF

Recently, a student of mine (Felix SĂŒĂŸenbach, now at the University of Edinburgh) and I published a little study on gaze-cueing, and how it is moderated by the trustworthiness of the gazing person.

In a nutshell, although instructed to ignore the gaze, participants shifted their attention into the direction where another person looked at (–> this is the well-established gaze-cueing effect), but more so when the sender was introduced as being trustworthy (–> which is the new result)

Bildschirmfoto 2014-10-30 um 09.32.39

 

We also found some exploratory evidence that the trait anxiety of participants moderates that effect, in a way that highly anxious participants did not differentiate between trustworthy and untrustworthy senders: Highly anxious participants always followed the other person’s gaze. For low anxious participants, in contrast, the gaze-cueing effect was reduced to zero for untrustworthy senders. (This exploratory finding, of course, awaits cross-validation).

Bildschirmfoto 2014-10-30 um 09.37.29

The paper, raw data, and R script for the analyses are on OSF.


 

SĂŒĂŸenbach, F., & Schönbrodt, F. (2014). Not afraid to trust you: Trustworthiness moderates gaze cueing but not in highly anxious participants. Journal of Cognitive Psychology, 26, 670–678. doi:10.1080/20445911.2014.945457

Publisher’s website: http://www.tandfonline.com/doi/abs/10.1080/20445911.2014.945457#.VEn3GVu17sI

Abstract Gaze cueing (i.e., the shifting of person B’s attention by following person A’s gaze) is closely linked with human interaction and learning. To make the most of this connection, researchers need to investigate possible moderators enhancing or reducing the extent of this attentional shifting. In this study we used a gaze cueing paradigm to demonstrate that the perceived trustworthiness of a cueing person constitutes such a moderator for female participants. Our results show a significant interaction between perceived trustworthiness and the response time trade-off between valid and invalid gaze cues [gaze cueing effect (GCE)], as manifested in greater following of a person’s gaze if this person was trustworthy as opposed to the following of an untrustworthy person’s gaze. An additional exploratory analysis showed potentially moderating influences of trait-anxiety on this interaction (p = .057). The affective background of the experiment (i.e., using positive or negative target stimuli) had no influence.

No comments | Trackback
  • Categories

  • Disclaimer: This is a multi-language site.

    This website is a conglomerate of old and new texts - some are in German, some in English. I do not have the time to translate each, so unfortunately this is a mixed-language site. I'm sorry. If you have any questions about the German texts, just write an email! Diese Webseite ist eine Mischung aus deutschen und englischen Texten - leider habe ich nicht die Zeit sie jeweils in die andere Sprache zu ĂŒbersetzen. Falls Sie Fragen zu einem englischen Text haben können Sie mir gerne eine Email schreiben!