Now I want to segue to Scott’s viral post, “An Open Letter To Steven Pinker On AI” (I am using an Archive link because his blog does not load well)in which he invokes the appeal to authority fallacy:
Meanwhile, Geoffrey Hinton, the Nobel Prize and Turing Award winning inventor of modern AI, quit his job to become a full-time activist warning about the possibility of superintelligence destroying the world. Yoshua Bengio, another Turing Award winning inventor of modern AI, also did that. Ilya Sutskever, the former Chief Scientist of OpenAI who helped invent ChatGPT and the modern LLM, wrote that superintelligence “could lead to the disempowerment of humanity or even human extinction”, and quit his job at OpenAI to found a company called Safe Superintelligence. Paul Christiano, who invented RLHF and headed safety at the US government’s AISI, says he thinks there’s a 20-50% chance AI kills everyone, and quit his job to start the Alignment Research Center. Sam Altman, Elon Musk, Dario Amodei, and Demis Hassabis – the latter two of whom are great AI researchers in their own right even aside from their business credentials – have all said or suggested that they think human extinction is a plausible outcome of their work. Katja Grace surveyed AI researchers who published at various important conferences and journals, and found that on average, her sample of 1,600 gave the proposition an 18% chance.
Experts are wrong all the time. See the failed predictions or policy on Covid (“flatten the curve”), 2008, Iraq, etc. Even Nobel laureates fall for it: Linus Pauling dabbled in medical quackery with his megadose vitamin C cancer claims. Luc Montagnier, who won the Nobel for isolating HIV, later promoted homeopathy-style “water memory” claims. Kary Mullis, who won for inventing PCR, denied that HIV causes AIDS. Brian Josephson pushed water memory and the paranormal. And of course, there was Paul Ehrlich, the late Stanford professor, whose 1968 prediction of mass starvation was not just wrong–he defended it until his death at 93 as the rest of the world moved on. If anything, more credentials seem to be correlated with worse forecasts or detachment from reality.
See my post, where it was 20-somethings, teen streamers, and “randos” on Twitter/4chan–without academic credentials, who watch YouTube and livestream all day–who were right about the tariffs being useless and Iran being a bad idea, whereas 40-plus-year-old podcasters with huge brands, credentials, and a multitude of followers “simping” for Trump were all wrong. The “nobodies” keep being right over and over. This makes sense when you think about it: people with no status and no credentials are looking at a problem with an empty or blank mind and have no conflict of interest.
This is not to say experts should be dismissed. But expertise, brand size, and name recognition are, on their own, terrible reasons to believe someone and awful predictors of accuracy outside very specific domains such as math or physics. There’s a big difference between proving a theorem and making a far-out forecast about what some technology will do.
Or from AI 2027, which is regarded as another authority figure of AI:
AI 2027 authors: “Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all.” … “very worried” we’re 3-12 months from AI takeover.
by
u/Puzzleheaded-King584 in
agi
Right now we’re almost at the end of 2026 and nothing has meaningfully changed economically or societally, aside from some math problems being solved. When 2027 ends and still nothing has meaningfully changed (aside from new iterations of Chat and Claude and more math problems being solved), are they going to backtrack? No way. They have already committed to the act. The issue is conflating different claims or moving the goalposts. I am not exactly accusing Scott of this, as his views are more nuanced than most, but consider the following:
1. AI will lead to jobs being lost.
2. AI will lead to net job losses, destroying more jobs than it creates.
3. AI intelligence (or what we call intelligence) is rapidly advancing.
4. AI is rapidly transforming industries.
5. AI will doom humanity.
6. AI is a bubble or is unsustainable.
I have seen people conflate many of these. A common argument is that AI will destroy jobs. But technology has always destroyed jobs. In any dynamic economy, jobs are always being created and lost. What matters is net job loss, which is a different and more specific claim. Moreover, AI becoming smarter or more capable does not imply #2 or #5. Going from “AI is really smart” to “AI destroys humanity” is a leap of faith, and heavy skepticism is warranted. But it’s assumed this should be self-evident, and God forbid anyone have the audacity to ask for an actual mechanism for how this happens.