I think the epidemic of irreproducibility as it pertains to the social sciences and success stories is way bigger than imagined. The survivorship bias is huge. It’s like you’re presented with what you are told is a blueprint (e.g., ‘grit’, ‘10,000 hours’, ‘mindfulness’, ‘growth mindset’, etc.) for success, persuasiveness, or anything else socially desired. What is ignored is everyone who follows said blueprint and fails.
If the reproducibility of social science findings or claims were held to even to even a small fraction of the of same standards as math or engineering, it would be a total failure. Yet people are making careers off this. It’s the intellectual/academic level equivalent of tarot card reading or astrology. The ‘growth mindset’, popularized by Carol Dweck, is possibly wrong yet it endures, similar to Angela Duckworth’s popularization of ‘grit’. There are probably a lot of financial interests–books, seminars, TED talks, careers–in keeping it alive.
From the above link:
What the team found was there is a correlation between someone having more of a growth mindset and doing well academically. However, the correlation is small and the findings do not support claims that growth mindset interventions have profound effects on academic achievement.
“We found limited evidence for mindset theory and the evidence varied by the specific claims,” says Brooke Macnamara, who co-authored the study. “For example, overall, we found a significant but weak relationship between growth mindsets and academic achievement, and a significant, but small effect of growth mindset interventions on academic achievement.”
True, success stories are not intended to be held to the same rigor as engineering, but the huge and hidden survivorship bias means that the success rate is indeterminate, because it’s impossible to know how many trials were run. All you know is that some people succeeded, but the denominator is unknowable. Incorrect ideas in the ‘hard’ sciences seem to be quickly buried, but they can last forever in the social sciences. It’s always possible to massage the data in a certain way or sample enough times to get the results you want. If you want a study that shows that grit is important, keep testing until you get the data you seek, and then write books and give talks about it.
Another problem is it’s impossible to make generalizations or anything precise in a scientific sense. Engineering and science succeeds because it can be reproduced, so the outcome is predictable based on some initial set of conditions. But then why do some people get much more milage from their ‘10,000 hours’, ‘growth mindset’, or ‘grit’ compared to others? Why do some succeed and others fail, even after controlling for many important variables? Often there are tiny subtleties that separate the successes from the near-misses, or it’s impossible to know. For example, what separates the 10 best golfers in the world from the top 100? Probably any differences in technique are too subtle that it would be impossible to replicate in a controlled setting or learned. Why did Wool and The Martian succeed so overwhelmingly, but many other self-published books also published at the same time barely sold at all?
Besides shoddy/cherry-picked research, as the so-called replication crisis has brought to light, I think the second problem is that the conversation is one-way. If you watch a TED talk or a Malcolm Gladwell book reading, often there is a Q&A at the end. However, are these questions screened in advance or filtered for ‘suitability’? Almost certainly. Also, there is no follow-up. Someone can ask a question, and the speaker can give a nonsensical answer or a non-answer, and not be forced to clarify what was otherwise a totally inadequate response. Public intellectuals are protected by artificial, safe environments that make them look good and shield them from effective, pointed criticism. To criticize someone like Paul Krugman and get his attention means you need a platform as big as his, which is impossible for most people.
That’s not to say it’s impossible, but journalists and public intellectuals tend to operate outside of peer review, such as publishing books intended for a general audience and op-eds instead of peer-reviewed papers. Peer review, for all its flaws, in theory provides a way to contest the veracity of any result in an impartial manner regardless of the size of one’s platform/brand. If someone publishes nonsense, you can bring that up with the journal editor. It doesn’t mean it will be remediated, but it will be considered.
The 2000 best-seller Arming America by former Emory University historian Michael A. Bellesiles was revealed to be a fraud after sleuthing by Clayton Cramer, an amateur historian who went public with his findings. He was initially rebuked by Bellesiles for not having sufficient credentials, such as a PhD. In 2002 after an independent investigation by Emory University was the extent of the academic maleficence uncovered. The book was revealed to be a complete hoax, including even fabricated quotes and data. This led to the resignation of Bellesiles in October 2002 and his Bancroft Prize being rescinded. Because the book affirmed what its reviewers wanted to believe, no one stepped back to ask if any of it was actually true.
As discussed in the post The Fake World of TED and Pop Psychology, a lot of behavior psychology, self-help, and other advice is predicted on the flawed notion that humans rationally respond to incentives, evidence, or ‘reason’. But the world is complicated, and human behavior is unpredictable. Such unknowns seldom yield to simple lifehacks or heuristics, whether it’s those Dale Carnegie, Napoleon Hill, Daniel Kahneman, or Seth Godin books, which sound good in theory until you see tons of examples of people who embody the opposite of those books, who are ‘antisocial assholes’, like in politics, the media, Hollywood, or even your boss, who are wildly successful. Of all the personal attributes that can explain the huge successes of Bill Gates, Elon Musk and Steve Jobs, ‘being good listeners’ is likely not one of them, and yet they are/were among the richest and most successful people in the world. Many people do not like Trump on a personal or visceral level, and there is nothing he can do short of dropping dead that would change those people’s minds (or the same for people who dislike Biden). How do you possibly use persuasion techniques or Dale Carnegie’s advice on those people?
Even studies as by Kahneman, which show that people act irrationally to incentives, like priming, can still fail to replicate or are not helpful, because the very idea of trying to assign order or to generalize the inherent irrationality of people, overall, makes it doomed to fail. If half the time people behave ‘predictably irrationally’, but also sometimes rationally, how is that useful? You cannot predict anything.
People are always looking for shortcuts or lifehacks to success, and there are no shortage of authors and researchers who promise that ‘one weird trick’ that will deliver results, because isn’t that what we all want, results? Yet it never seems to work well. Those tips and tricks that sound really convincing in video, article, or book format, when put the ultimate test, that being real life, tend to fall way short. The problem is, life hacks are are another form of arbitrage–that is–buying low and trying to to sell high. This works is only a few people are doing it in a large market, but when everyone is using the same tricks, they tend to stop working. This means that things such as ability, talent, and skill and other quantifiable things matter more, not that pre-suasion technique you read at an airport bookstore.