The Social and Intellectual Context of AI Doomerism

People who speculate about artificial minds have a thought experiment: if you lock a superhuman intelligence in a box, with just a way to ask it questions and a way for it to send back the answers, how do you stop it from persuading someone to let it out? Today some people who read the right parts of the Internet ten years ago are afraid that some terrible ideas have escaped geeky online communities and are commanding money and policy in the wider world. Outsiders don’t have the background knowledge to know why this is a bad idea. But a lot of the criticism is hyperbolic, very personal, and mixes unverified claims with matters of public record. Just below the surface are such baroque ideas and cycles of interpersonal relations that it is exhausting to learn what happened, disturbing to think about it, and hard to explain why this matters to anyone but a few very clever, very strange people who spend a lot of time on the Internet (and maybe social media these days). I found one series of essays that may help.
If you are interested in the implications of technologies like ChatGPT and OpenAI, but you did not know that Nick Bostrom the philosopher, Scott Alexander the psychiatrist, Eliezer Yudkowski the autodidact, Wei Dai the bitcoin developer, Caroline Ellison the fraudster, and Peter Thiel the wannabe Lex Luthor are no more than one degree of separation from one another, or did not know words like “singularity” “MetaMed” and “Rocco’s Basilisk”, I suggest you read: Jon Evans, Extropia’s Children (2022) and maybe Maciej Ceglowski’s talk Superintelligence: The Idea That Eats Smart People (2016). Evans avoids unnecessary personal attacks and gossip about people’s private lives beyond documenting the social and intellectual connections.

Whatever you think of these ideas and new machine learning and language models, they come from a unique group of people with peculiar ways of thinking and a tendency to fall into the same bad ideas[1] and patterns of behaviour. If you evaluate them de novo, you will miss a lot of essential context. I just have an undergraduate degree in computer science, so the Internet does not need my thoughts on chatbots and spicy autocomplete. But I am vaguely aware of some of these people, their connections, and people I respect who skewered earlier versions of their ideas. Jon Evans’ essay is a good summary.
PS. If you are trying to organize what you remember about these ideas before 2022, it may also help to know that Émile P. Torres seems to be the writer who used to go by the name Phil Torres. On twitter Torres’ pronouns are they/them.
(scheduled 6 June 2023)
[1] yes, among those “bad ideas” is the belief in a racial hierarchy of IQ with white Europeans near the top and black Africans at the bottom; Alexander (again in 2025!), Bostrom, and Gwern have explicitly endorsed these theories at one time or another, and other figures in the movement just post a lot about IQ and g and genetics which in my experience is a warning sign. Even if you exclude the ‘scientific’ racists who used to hang around Alexander’s blog as ‘no true rationalists’, I don’t think there is room to doubt that some major figures in these spaces are fascinated by the idea that racial stereotypes might be scientific truth. I am using race-and-IQ as an example because two three major figures in the community have endorsed it in writing and I think my gentle readers can agree that its a very bad sign, whereas other bad ideas are harder to document or difficult to explain in a short post.
Edit 2023-06-14: refreshed my memory of some of his essays and decided what stance gwern takes on race and IQ (in a review of a book on a sperm bank of Nobel prize winners!) I want to honestly represent people’s public statements without either sensationalizing or pretending I can’t hear a whistle which makes every dog in earshot howl. (NB. that gwern attributes some facts to an Emil O.W. Kirkegaard who is not the existentialist philosopher – yet another person in this network who is not just an eccentric intellectual). If you want an assessment of these ideas by an expert outside the rationalists, please go ask an anthropologist not an ancient historian and philologist!
Edit 2025-01-17: added another link to someone in these circles pushing race-and-IQ and citing less respected Internet personalities
I’m going to trust AI to protect me from UFOs. 🙂
Or vice versa.
Who is to say that the UFOs were not sent by the future AI using acausal trade to stop us from stopping SkyNet? Or were they sent by the AI which is eating the Magellanic Cloud so we don’t develop SkyNet and let it eat our resources before Magellan AI can colonize the Milky Way?
I don’t know but I bet if you locked me in a room with an Internet connection and a lot of pop science books I would have thoughts after just a few years.
[…] tend to be concentrated in specific cities and subcultures such as Old Media journalism and the rationalist movement, I don’t live in one of those cities and those cultures are not my culture (Dan Luu above has […]