Leftovers from Last Time

My last post on Tom Chivers’s book The AI Does Not Hate You took an embarrassing amount of time and work to get done. My posts on books tend to, because I jot down a lot of thoughts when reading and then I have a hot mess to take care of, which I put off forever. Many such thoughts overlap a little bit, some are interesting but tangential, and some work together but point in different directions and thus become difficult to put in sequence in a way that feels natural.

Let’s say my notes after The AI Does Not Hate You looked something like this:

Not a nice straight line. A bunch of dots, some larger than others, to be connected somehow. I spent a lot of time and effort trying out different ways to arrange them. Most weren’t particularly elegant.

Several awkward twists and turns.

Even worse somehow.

The phrase “kill your darlings” applies. Sometimes that one part you love so much ruins the flow of the piece and you have to get rid of it for the rest to work. For me there’s also the problem that, while I often complain about not being productive enough I also often end up writing way too much[1]. Getting the review post down to a reasonable length required some savagery. In the end it came out something like this:

There’s quite a lot left here, after the main narrative had been removed. This post is me picking up those leftovers and throwing them into the frying pan in the hope of a decent meal. Besides the main story in the last post (orange, below), some stuff just got thrown into the trash (light gray), something else got big enough to be put into its own, forthcoming, post (blue), and the rest (green) is here.

It probably makes sense to read the main piece first, unless you already have.

•

The art of the mind dump

The first tangent is relevant to the intro. It’s on Chivers’ explanation of why rationalist movement founder Eliezer Yudkowsky wrote many hundreds of thousands of words on his view of what rational thinking is just because he wanted people to understand exactly why he thinks what he thinks about artificial intelligence:

So in order to explain AI, he found he had to explain thought itself, and why human thought wasn’t particularly representative of good thought. So he found he had to explain human thought–all its biases and systematic errors, all its self-delusions and predictable mistakes; he’d found a natural home on Overcoming Bias. And to explain human thought, he found he had to explain–everything, really. It was like when you pull on a loose thread and end up unravelling your entire favourite jumper.

Personally I appreciated this “omni-philosophy” aspect of Yudkowsky’s writing much more than the specific conclusion he wanted to get to. The “this is my whole worldview” genre is fascinating. You get to look straight into the mind of another person and see how things hang together in there.

I very much understand why he felt the need to do what he did, because I have the same feeling. In my case explaining my entire worldview feels necessary for really, truly, communicating why I think what I think, not about AI, but disagreement.

I kind of want everyone to do this. If we had to read a hundred thousand words of somebody’s philosophizing before feeling confident that we understand their point of view, the quality of communication on complex topics would greatly improve. Sadly it’s ridiculous, so the closest serious exhortation along those lines I’ve done is 30 Fundamentals.

Writing these mind dumps is a lot easier if you’re like Yudkowsky and can, seemingly, write down your thoughts as you’re thinking them and have it come out making sense. I’ve heard the other major rationalist writer Scott Alexander say that he just writes what he’s thinking and doesn’t quite understand why people say writing is so hard. I’m unbelievably envious. For most of us, pulling on a thread and actually winding up with a thread instead of a curled-up monster takes a lot of hard work. I tried to go free-wheeling last year and I didn’t particularly like the result.

•

From weirdness to badness

There’s a part of The AI Does Not Hate You when Chivers discusses how the weirdness of the rationalist community puts people off. The cuddling and widespread polyamory leads to accusations of being a sex cult, and I understand how you can come to that conclusion if you take a particularly leering look at the whole thing. According to the book, a Reddit community dedicated to mocking the rationalists “latched on with enourmous glee” to an old OKCupid profile Eliezer Yudkowsky had put up, where he was (as he often tends to be) self-aggrandizing, very open about his personal fetishes, and displaying an open, straightforward attitude to sex. Chivers talks about his own reaction to it:

It gives me an icky sensation, but again, it’s just a bit weird, not wrong, at least as far as my moral framework goes. If women want to message him, great; I hope it works out for all parties. But for some people the ‘ick’ factor is stronger, I think. I’ve heard it described as Yudkowsky ‘trawling for sex’, but I can’t see how you get there. He’s just openly stating that he wants sex and that if anyone wants it with him they should ask.

I saved this quote because I think it’s important. We all have “ick” impulses. That’s fine, it’s part of being human. But I very much object to treating your ick-reactions as valid moral judgments. There’s a lot of history of people doing that, and the vast majority of it abhorrent from the viewpoint of contemporary morality[2].

Chivers also notes this weird-to-immoral mental slide:

I don’t think the Rationalist community is a sex cult. But people on the outside, those of us like me in our hetero-monogamous, married-and-settled-down, two-kids-and-a-people-carrier world, find their whole thing deeply weird, and for us it is very hard to separate ‘weird’ from ‘immoral’.

I guess this is a personality thing, because I don’t think I have this impulse. I do notice weirdness (well, most of it) of course, but I feel no particular pull to translate it into derision or moral condemnation. The exception is when weirdness comes off as inauthentic or performative. I do judge that, but not because weirdness is upsetting as such. I just find such behavior immature, uninteresting and tiresome. I get no sense at all, from any part of the rationalist community, that their weirdness is a performance.

One of my favorite things about the community is its full-throated acceptance of by-standards-of-social-normality weirdness. It is understood and accepted that just because an idea is unusual, socially undesirable, or held by off-putting people, it does not mean that it’s wrong.

Jacob Falcovich of Putanumonit tells this story:

I told my coworker that when I’m in a rush and need a measured dose of caffeine I just chew on a handful of roasted espresso beans. The taste is not actually bad — if you like chocolate covered beans, you may not totally need the chocolate. I suggested to my coworker that she try it at least once, just to know what it tastes and feels like.

She adamantly refused, citing “that’s not how it’s consumed” and “it’s weird, people don’t do that” as her main objections. I countered that these are facts about people, rather than facts about coffee beans. While you can infer some things about beans from observing people, the beans are right there in the office kitchen to be experienced directly. My coworker seemed unable to grasp the distinction, treating the social unacceptability of eating coffee beans as akin to physical impossibility.

I know I shouldn’t judge this person, but I have to summon a surprising amount of mental generosity to avoid it. I guess I carry some resentment towards “weirdphobia”. And I’m not even that weird! I live a very conventional life, with a wife and 2.1 children, two sensible cars, a mortgage, a conservative wardrobe and a job with a lot of powerpoints. My gut just hates that attitude for some reason. Part of me sometimes want to grab people[3] and say no this isn’t weird, you are weird! You just don’t know it, because the rest of humanity is sharing, enabling and validating that weirdness!

We humans are all weird. We’re packed to the brim with unexamined assumptions and black-box concepts that don’t make sense when we look into them, but we’re nonetheless convinced they’re well supported and in no need of explanantion. And we react negatively to people challenging, questioning, or just not sharing them.

Weirdness and politics

When we enter politics this gets a lot worse. Wooo-boy. Here “is true/makes sense” is conflated with “is socially accepted/desirable” all the time like everywhere else, but the stakes are extra high.

Weirdness, in the sense that describes the rationalists, means being very aware of this disconnect. One consequence is that they accept an unusually wide range of political beliefs. In The AI Does Not Hate You, Chivers tries to deal with this, but it’s far from the book’s strongest part. I think he shies away from really putting the spotlight on the aforementioned disconnect, and without that I don’t think you can explain the rationalist attitude to political discussion and debate very well.

There are two chapters on politics, specifically about feminism and the neoreactionaries. I believe Chivers is aware he needs to discuss them but doesn’t quite want to and aims to keep it as short as possible. I understand why. If given anything more than the bare minimum, they could easily swell up to overshadow the whole rest of the book and that wouldn’t be good for anyone. I might have done the same thing myself if I wanted the parts about AI and existential risk to take center stage.

In the review I wrote a bit about the problems with the feminism chapter and I could write a lot more, but I’ll save that for another post. I’ll just mention how it ties into the disconnect described earlier: feminism, insofar as it is a complete ideology like conservatism, liberalism, communism, capitalism, anarchism etc. has a lot of weaknesses and blind spots (as all ideologies do). However, in polite society those weaknesses aren’t pointed out as much and as clearly as they could be. Dissent exists, sure, but it’s mostly expressed as silence, lack of interest or total but superficial rejection, not detailed criticism. It’s seen as attacking women, which is very bad form, according to both feminism and to traditional gender norms.

Exploiting this norm protects ideological feminism from intellectual criticism from everyone with social capital to safeguard through exactly that thing that makes people confuse “is socially accepted” with “is true/makes sense”. “Weirdos” don’t see such norms as sharply, and are more likely to object to things that don’t make sense to them.

In short, the conflict isn’t the contingent result of some particular event. It’s inherent, because curious, intelligent weirdoes are the natural enemies of belief structures that rely on social acceptability to not get criticized.

And the neo-who?

Rationalists’ acceptance of weirdness can push them in all kinds of political directions. Some follow their compassion towards an extreme concern about suffering that go beyond everyday charity giving, Effective Altruism or veganism into asking whether fundamental particles can suffer. A few other affiliated people have gone in the other direction, like the neoreactionaries.

I don’t know a lot about them. As far as I know they’re a tiny fringe group who want to abolish democracy, roll back cultural modernity and return to a sort of feudal/monarchic politics. They embrace hierarchy and readily incorporate innate inequalities into their political vision.

Something about Chivers’ discussion of them bothered me. They’re extreme, sure, but there are milder, softer versions of some of their beliefs that are more reasonable. They have some valid criticisms of contemporary society the same way communists and anarchists do. I got the feeling that he uses this very small group as a scapegoat, as a tool to explain away any beliefs disproportionately accepted among rationalists but sits uncomfortably with center-left-polite-society norms, such as the importance and high heritability of intelligence, skepticism against victimhood narratives, etc. That is, he makes the rationalists more palatable to the polite society he’s speaking to by suggesting that the relatively larger presence of such beliefs is only because of a small number of unusually talkative assholes. It therefore doesn’t suggest that very same polite society is genuinely and importantly different from the rationalists in ways they ought to examine, for example in confusing social acceptability with correctness.

Sure, I get it, but it feels ever so slightly cowardly. He does explain that rationalists take almost all beliefs as worth seriously engaging with, even bad ones. But in my interpretation he doesn’t reject the implication that they’re just humoring people as a pastime or temperamental quirk, instead of seriously doing what everyone else should be seriously doing as well. Fair or not, it seems to me that Chivers excuses this behavior more than he defends it, which I honestly think it deserves.

My life in conservatism-free society

I probably need to go into some personal background to explain why this feels important to me, and why I carry just a bit of a grudge against what I called “center-left polite society”.

To begin: I consider the political left and right wings to be on roughly equal footing, morally and intellectually. They’re both partial narratives that each capture different aspects of the physical and moral universes. Together they form something better than apart, and any cultural milieu that one of them dominates too forcefully invariably deteriorates, like a muscle that faces no resistance. I don’t think this is sufficiently appreciated in much of intellectual culture, where merely calling something “right wing” constitutes an accusation.

The society in which I grew up was almost entirely conservatism-free. The political right in this country never really pushed any genuinely conservative argumentation that I can remember; what emerged from its collective mouth is better described as neoliberal. Deregulation. Lower taxes. Actual conservatism was something you’d see on TV from the United States, and it was weird and alien, proof of how crazy they were over there and what an exotic society they had.

As I understood the world when I grew up and throughout early adulthood, conservatism was nothing but another word for anti-intellectualism, blind stubbornness and wild irrationality. I was over 30 years old the first time I came across an article that made conservative points in a reflective, rational, and intelligent manner. It felt very strange.

I’m much more used to it now. Has it made me a conservative? No, it has not — not unless you use it to mean just “non-radical”, but that I always were — but it has convinced me that conservative views has to be respected and engaged with. Say, given about 20-30% credence, as opposed to the current -50% (actively derided). The thoughtless smugness that thinks of itself as enlightenment that I used to find normal and obvious when I was younger now bothers me. As a result, the center of my political philosophy now is that we shouldn’t feel too confident in the correctness of our opinions.

The compulsive balancer

During this coronavirus crisis me and the wife both work from home, and we discuss the pandemic a lot. She’s very anti-lockdown, and opposed to almost all restrictions other than keeping retirement homes protected. I’m sympathetic to her points but much more unsure, and it drives her nuts that I keep trying to (repeatedly, she doesn’t listen) explain the reasoning of the pro-lockdown side instead of just agreeing with her. “Just tell me your own opinon, not what anyone else says!” she asks, not unreasonably. But I don’t have a strong opinion on virus mitigation strategy to give her. There are too many unknowns. I do have a strong opinon on how people express their opinions. When people overconfidently dismiss opposing arguments as nonsensical and irrational, and I know they aren’t, I want to push back, no matter what I personally think. That we should be balanced and understand opposing views is what I personally think.

It is for this reason I tend to feel protective of conservatism (for example, see here). I get irritated when it doesn’t get a fair hearing among intelligent, thoughtful people, because I expect more from them.

There’s an illustrative passage in The AI Does Not Hate You. In defending the rationalists against the accusation (sic) of being right-wing, Chivers points out that many of them spoke out against Trump. That’s true, and [BEGIN INCANTATION] I don’t like him nor would I vote for him or someone like him [END INCANTATION], but I also don’t like that I feel obligated to say so. I don’t like that it often goes without saying that supporting either him or Brexit or whatever it may be makes people wrong and bad by default and therefore what they say can simply be dismissed without examination. There’s just… enough of that in the world already. Take other people seriously. Very few are wrong about everything[4]. And even when somebody expresses a dumb idea, and God knows many do, there’s often a smarter version worth taking seriously.

•

Can you actually reason yourself into changing?

The “I must save the world” impulse that Eliezer Yudkowsky seems to have, and has had for his whole adult life according to The AI Does Not Hate You, is odd to me. The thinking is on full display in his epic fanfic Harry Potter and the Methods of Rationality, where Harry also feels obligated to take the world’s problems on his shoulders[5]. And sure, I understand it perfectly well on an abstract, logical level. I just don’t and can’t share this moral urgency about all the suffering in the world and the dangers humanity faces in the future, no matter how sound the argument might be. I have just enough of that normal human capacity to push it all out of my mind.

I get how frustrating it must be to Yudkowsky and others to encounter this all the time, it’s probably the source of that “everyone is insane but me” sense I’ve sometimes gotten from him. He’s right and we are, in a way, as I’ve said.

But this insanity can be protective. Most of us fear trusting our abstract reasoning when it takes us to strange places, and that’s often a wise impulse: hubris is a staple of literature for a good reason. Trusting your own thinking over tradition has probably been a bad idea throughout history when the knowledge available to reason from has been patchy and mistakes frequently lethal. I probably have less of this self-distrust than most people do, but Yudkowsky appears to have little to none.

Perhaps his sense of urgency is reinforced by the actions it suggests. If a line of moral reasoning suggested that I must dedicate my life to tasting every flavor of ice cream in the world, that would probably make it easier for me to accept than if it suggested that I should work around the clock at a job I hate and donate all my earnings to charity. According to Yudkowsky’s reasoning, he and many who agree with him have important and interesting things to do in saving the world. Chivers says:

At the risk of overgeneralising, the computer science majors have convinced each other that the best way to save the world is to do computer science research.

I don’t mean to call Yudkowsky dishonest. I don’t think he is. But it’s also true that his conviction that AI alignment is the most important thing ever enables him to work with stuff he likes and is good at. That’s not the case for me, and that’s likely part of the reason I can’t think of it as the most important thing in the world the way he does.

The other option is that some people really can reason themselves into changing their priorities on a deep level[6]. I have a friend somewhat like that. If he convinces himself that something is the rational or morally correct thing to do, he does it, whether it’s subsisting on nothing but carrots and Brie, moving over 1000 km to the municipality with the lowest tax rate or travel around the world on boats and trains because flying hurts the environment. He lacks the “wait, no that’s crazy” module most people have. Half of me admire him and the other half is a little freaked out.

I should use that freaked-out feeling. I should use it to model how normal people must think of rationalist-type weirdoes. Perhaps it even explains the weird-to-immoral slide: people like this are unpredictable and cannot necessarily be trusted to follow convention and do what is expected of them, which easily translates to “cannot be trusted” in general.

We ought to look for opportunities to use our feelings towards X to understand how others feel about Y more often. Call it an “empathy pump”.

•

The selective conservatism of Star Trek

In the review I talked about how I’ve become uncomfortable with speculation about the far future. Isn’t that a little strange for someone who likes science fiction, as I always have? I think the key point is that most science fiction avoids doing away with recognizeable human beings, for obvious reasons. A real future where that happens is a whole different ballgame.

Case in point: Anyone who’s read a lot of this blog will know from the occasional reference that I’m a Star Trek fan. And Star Trek is, for all its mundane political progressiveness, distinctly conservative about humanity. It rejects both genetic engineering and technological mind-altering. And while it embraces very limited artificial intelligence it’s carefully portrayed as not being massively transformative.

Star Trek has fantastical technologies but leaves us humans remarkably intact — and notably always in charge of our own destiny. In fact, our self-determination is a central theme across the franchise. The original series in particular, on several occasions, comes out decidedly against things like friendly AI and the whole idea of us being taken care of by superior beings, as if we were children or pets[7].

Star Trek can even be called human-chauvinist: humans are special. We’re special because of our versatility, capacity for self-improvement and exploratory spirit. It preaches acceptance of others, certainly, but only to a point — you’re still expected to fit in. It has a staple of characters who strive to grow by becoming more human-like, instead of cultivating their difference and possible superiority. Yes, Star Trek portrays a bright future, but that’s only because it avoids future shock by populating it with people like ourselves.

•

AI — the spaceflight of the 2020’s

Star Trek’s vision of the future originates in the 1960’s and seems odd and quaint today, given how technological development has progressed. Computing, information and communication technology is ahead of what was imagined back then, but our progress in spacefaring has been anemic compared to the expectations of two generations ago. Just look at the 2001 movie — that kind of capability was supposed to be 20 years out of date now.

From my future-shocked perspective this actually offers some hope. Spaceflight was all the rage in the 1960’s because we made great strides then, but it turns out that the higher-hanging fruit is quite a lot higher up than we first thought. We won’t be building a starship Enterprise any time soon, or ever, for so many technological, practical and economic reasons. AI is similarly all the rage right now because we’re in the middle of making big advances. Will that slow down too? Is imagining a future full of AI the contemporary equivalent of imagining a future full of starships in the 1960:s?

Maybe. Maybe strong AI:s are theoretically possible, but we find out that the problems are similar to the problems with spaceships: it’s going to be completely impractical, pointless and uneconomical to build and use them. So we don’t. Instead we use an army of mundane, specialized and limited AI:s.

I feel like a party pooper but it might be the outcome I prefer, if I’m honest with myself. Sorry, visionaries.

• • •

Notes

[1]
That’s how two simple-seeming ideas turned into an 11000-word treatise on what racism is and an 8000-word two-parter about the political compass (part 1, part 2).

[2]
And if I understand things correctly, the particular anti-rationalist community in question is mostly on the political left, which means that they, especially, should be above this sort of thing, given its history.

[3]
By that I mean “people” in the abstract, not specific people.

[4]
I make absolutely no claim that the right wing doesn’t do the same sort of thing. They definitely do. It’s just a lot easier to ignore them. They don’t have the same cultural power, dominant position in the well-connected intelligentsia, and presence in the social class in which I exist. Put me in a different cultural context though, and I may well come out with some very different pet peeves.

[5]
This thinking also comes through in some of Scott Alexander’s writing, especially the Comet King character in Unsong.

[6]
I have changed my mind on things over the years of course, but typically not by formal reasoning. It’s mostly a matter of, for one reason or another, starting to see the world from a different perspective and reinterpreting past knowledge  and experiences. It doesn’t feel self-directed nor logical exactly.

[7]
Isaac Asimov’s robot novels that leads into the Foundation stories shows a similar sensibility by having us reject life extension and dependence on robots because it undermines our human spirit.

Did you enjoy this article? Consider supporting Everything Studies on Patreon.