About that metaphor priming paper…

I recently reported this paper for impossible means [link]. A few notes about the process and paper:

I was very pleased with how Patricia Bauer, the editor of Psych Science handled everything. She was super appreciative and kind. And Greg Francis did an excellent job on the analysis and writeup I thought. I have zero complaints about the process. They also handled everything very quickly. 

I would like to give more context about this paper. 

  1. Quentin Andre hit the nail on the head by pointing out the effect size issues [link]. d=1.3 is not super likely with a metaphor priming study. 
  2. Some of the means are impossible. Either there were unexplained errors or the summary statistics were generated without any participant data. To be clear, I didn’t expect to see any participant data and none was provided by the authors.
  3. This paper has two replications. The first replication effort (Firestone & Scholl 2014) was marginally successful but the authors claimed the effect was due to a confound. The second replication effort (Brandt, IJzerman, & Blanken 2014) I found to be more credible than the first and it failed. 

The second and third author have some history:

  1. Chatterjee/Sinha retraction
  2. Sinha Expression of Concern

And that is all I have to say about it for now…

Comments

2 responses to “About that metaphor priming paper…”

  1. Greg Francis Avatar

    I don't think we should have much faith in the Firestone & Scholl (2014) replication. It is part of a set of experiments that seem "too good to be true". Details at https://www.cambridge.org/core/elements/abs/hypothesis-testing-reconsidered/B21AC3363858E507A47571C020038F40

  2. Aaron Charlton Avatar

    Okay, wow! Great analysis. I didn't look at it as closely as you have but my initial sense was that all or most of the studies they replicated seemed to be unreplicable. I thought maybe they were trying a bit too hard to get them to work so that they could show off their cool tool for detecting confounds. Now reading your analysis, it does seem like that is in fact what happened.

Leave a Reply

Your email address will not be published. Required fields are marked *