Approximate replication rates in psychology:
social 37%
cognitive 42%
personality 55%
clinical 44%
The incentive of all psychology researchers is to do new work rather than replications. Because of this, publicly-funded psychology PhDs should be required to perform study replication as part of their training. Protocol + results should be put in a database.
IIRC the 2013 "racism predicted by telling leading questions" one and ita predecessor, which is listed here but also says there is slight trend toward replication, is just based on implicit association tasks. So you have a green and red button for good and bad, and then a word pops up and you have less a second to choose which button to press. Oversimplifying complex thought processes in my opinion is junk psychology.
> Smile to Feel Better Effect
> Claimed result: Holding a pen in your teeth (forcing a smile-like expression) makes you rate cartoons as funnier compared to holding a pen with your lips (preventing smiling). More broadly, facial expressions can influence emotional experiences: "fake it till you make it."
I read this about a decade ago, and started, when going into a situation where I wanted to have a natural smile, grimacing maniacally like I had a pencil in my teeth. The thing is, it's just so silly, it always makes me laugh at myself, at which point I have a genuine smile. I always doubted whether the claimed connection was real, but it's been a useful tool anyway.
> Source: Hagger et (63!) al. 2016
I can't help chuckling at the idea that over 1.98 * 10^87 people were involved in the paper.
Disturbing fact: The Stanford prison experiment, run by Philip Zimbardo, wasn't reproducible but that didn't stop Zimbardo from using it to promote his ideologies about the impossibility of rehabilitating criminals, or from becoming the president of the American Psychological Association.
The APA has a really good style guide, but I don't trust them for actual psychology.
>Source: Stern, Gerlach, & Penke (2020)
Wow, what are the odds?
https://en.wikipedia.org/wiki/Stern%E2%80%93Gerlach_experime...
Little of this is considered cognitive psychology. The vast majority would be viewed as "social psychology"
Setting that aside, among any scientific field I'm aware of, psychology has taken the replication crisis most seriously. Rigor across all areas of psychology is steadily increasing: https://journals.sagepub.com/doi/full/10.1177/25152459251323...
> Claimed result: Adopting expansive body postures for 2 minutes (like standing with hands on hips or arms raised) increases testosterone, decreases cortisol, and makes people feel more powerful and take more risks.
A heuristic I use that is unreasonably good at identifying grifters and charlatans: Unnecessarily invoking cortisol or other hormones when discussing behavioral topics. Influencers, podcasters, and pseudoscience practitioners love to invoke cortisol, testosterone, inflammation, and other generic concepts to make their ideas sound more scientific. Instead of saying "stress levels" they say "cortisol". They also try to suggest that cortisol is bad and you always want it lower, which isn't true.
Dopamine is another favorite of the grifters. Whenever someone starts talking about raising dopamine or doing something to increase dopamine, they're almost always being misleading or just outright lying. Health and fitness podcasters are the worst at this right now.
Before dunking on psychology for not replicating, remember this is a cross-discipline problem.
In biomedicine, Amgen could reproduce only 6/53 “landmark” preclinical cancer papers and Bayer reported widespread failures.
Is anyone tracking how much damage to society bad social science has done? I imagine it's quite a bit.
famous cognitive psychology experiments that do replicate: IQ tests
http://www.psychpage.com/learning/library/intell/mainstream....
in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.
claimed result: Women are more attracted to hot guys during high-fertility days of their cycles
> Claimed result: Women risk being judged by the negative stereotype that women have weaker math ability, and this apprehension disrupts their math performance on difficult tests.
I'll never understand stances trying to hide biological differences between different sexes or ethnic backgrounds.
We know for a fact that sex or ethnicity impacts body yet we seem unable to cope with the idea that there are also differences in how brains (and hormones) work.
Women have, on average, a higher emotional intelligence which is e.g. tied to higher linguistic proficiency. That helps in many different fields and, on average, women tend to learn languages easier than men.
At the same time, on average, they may perform slightly worse than men in highly computational fields (math or chess).
I want to iterate what I'm getting at to before the rest of the post:
Genetics matter when you look at very large samples, but they are irrelevant on smaller (or single) samples.
I feel NBA provides a great example.
On average, african americans are taller than white men and have a higher muscular density.
On large samples, they tend to outperform white men. But as soon as you make the samples smaller, even at elite levels, you find out that Larry Bird (30+ years ago) or Nikola Jokic (today) are the best players in the world.
Same applies to women, just because average samples will explain some statistics, such as on average females performing worse on maths, won't change that women can be the best chess players or cryptographers in the world.
Given how long the whole field has been malicious/incompetent failing at basic statistics and sweeping it under the rug, I think that rather then discarding the experiments that don't replicate the better baseline is to discard everything and wait for the future generation of better cognitive psychologists that come up with any good discovery that is widely replicated?
All the "hypothesis" or supposed "results" are so bonkers than it's an insult to intelligence itself that such things can be "proved" with psych "experiment".
Not that it matters, most of the psychology field is inherently bullshit, those are just the example of cases they went so far in the insult to intelligence, no amount of "studies" and rhetoric can save them.
Dear lazyweb: is there the opposite of this list anywhere?
No mention of the Stanford Prison Experiment I notice.
Papers should not be accepted until an independent lab has replicated the results. It’s pretty simple but people are incentivized to not care if it’s replicable because they need the paper to publish to advance their career
The economics of pyschology are the psychology of economics.
If you won't trust the process, you will gain no real outcome.
What we recieve from the process is not necessarily tangible, but instead a fresh perspective on what may be possible. Thus, the inversion is complete, and we may then move forward.
i wonder the replication rate is for ML papers
Well, at least the growth mindset study is not fully debunked yet. It's basically a modern interpretation of what we've known to be true about self-fulfilling prophecies. If you tell children they are can be smart and competent if they work hard, then they will work hard and become smart and competent. This should be a given.
If the "failed replication" was a single study, as in many cases listed here, there is still an open question as to whether the 1) replication study was underpowered (the ones I looked at had pretty small n's), or 2) the re-implementation of the original study was flawed. So I'm not so sure we can quickly label the original studies as "debunked", no more than we can express a high level of confidence in the original studies.
(This isn't a comment on any of the individual studies listed.)
> Most results in the field do actually replicate and are robust [citation needed], so it would be a pity to lose confidence in the whole field just because of a few bad apples.
Is there a good list of results that do consistently replicate?
One thing that confuses me is that some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.
The ego depletion effect seems intuitively surprising to me. Science is often unintuitive. I do know that it is easier to make forward-thinking decisions when I am not tired so I dont know.
Now I want to know which cognitive psychology experiments were successfully replicated though.
I thought we knew that these were vehicles by wannabe self-help authors to puff up their status for money. See for example “Grit” and “Deep Work” and other bullshit entries in a breathlessly hyped up genre of pseudoscience.
Note that nearly non of these studies are pure cognitive psychology. Most have intersections with social psychology (and I would deem primarily social psychology) or developmental psychology. For example the debunked study on social priming was published in Journal of Personality and Social Psychology.
This title would be much more accurate if the author omitted “cognitive” from the title.
A key factor behind psychology's low replication rate is the absence of theories that define the field. In most science fields, an initial finding can be compared to theory before publication, which may weed out unlikely results in advance. But psychology doesn't have this option -- no theories, so no Litmus test.
It's important to say that a psychology study can be scientific in one sense -- say, rigorous and disciplined, but at the same time be unscientific, in the sense that it doesn't test a falsifiable, defining psychological theory -- because there aren't any of those.
Or, to put it more simply, scientific fields require falsifiable theories about some aspect of nature, and the mind is not part of nature.
Future neuroscience might fix this, but don't hold your breath for that outcome. I suspect we'll have AGI in artificial brains before we have testable, falsifiable neuroscience theories about our natural brains.
> Claimed result: Listening to Mozart temporarily makes you smarter.
This belongs in a dungeon crawl game. You find an artifact that plays music to you. Depending on the music played (depends on the artifact's enchantment and blessed status), it can buff or debuff your intelligence by several points temporarily.
This is a great list for people who want to smugly say "Um, actually" a lot in conversation.
Based on my brief stint doing data work in psychology research, amongst many other problems they are AWFUL at stats. And it isn't a skill issue as much as a cultural one. They teach it wrong and have a "well, everybody else does it" attitude towards p-hacking and other statistical malpractice.