Nick Bostrom – Grey Patterson

Nick Bostrom and Milan M. Ćirković

I’d love to see an updated version of this book, because some of the chapters held up really well, a couple need some revised timeline estimates,¹ and a couple would be heavily revised in light of how the last couple years have gone.²

The gist of it is, there’s a category of potential problem that can bring human civilization to its knees… or wipe out all life on earth. Some of them, we’re working on; some, we’re powerless to stop; and some, we’re actively making worse.

After establishing that core concept, the book splits up; each chapter is by a different author or authors, and they each address their own area of expertise. A volcanologist addresses the possibility of a super volcano eruption; an astrophysicist talks about the likelihood of gamma ray or cosmic ray bursts.³

Being the kind of person that I am, there wasn’t actually much in this book that was a new concept to me. This is the kind of thing I think about all the time! And so, given that background, it was a surprisingly uplifting book. Aside from the “the universe decided there isn’t life on earth anymore” type issues, every chapter came with advice on how to prevent or mitigate the issues, and discussion of who’s already working on it.

None of the problems discussed are solved, or anywhere near. I’d say that the one that’s currently in best shape is either naturally-occurring pandemics or nuclear proliferation—we have reasonably robust institutions, within the UN, working on those. Building a nuke is frighteningly easy, but getting the raw materials is, fortunately, very difficult. New diseases keep cropping up, but we’re getting reasonably good at developing vaccines, and things with sufficient lethality to totally collapse human civilization kill to fast to spread that well.

The other issues, though? Well… they remain a work in progress. AI researchers are still playing with fire, and CRISPR CAS-9 has made the possibility of engineered pandemics terrifyingly real.

All told, I’d call this book “required reading for anyone working at the UN.” And every world leader. It’s a long read, but the chapter divisions make it fairly digestible; I do recommend it, though with a caveat of “maybe not if you’ve been suffering from anxiety.” It… isn’t likely to help with that, unless you, like me, are already anxious about all these things. Give it a go.⁴

I can report that we didn’t crack self-replicating nanotechnology by 2020, for example. ↩
Although, actually, the one on dealing with pandemics was pretty much spot-on for what should have been done; the revised version would probably include a lot more pointing at our new historical counter-examples. ↩
The latter being, by my standard, among the scariest concepts in the book. No way to see it coming, and nothing we can do about it regardless. To borrow a term, it’s an out-of-context problem. ↩
This is a Bookshop affiliate link – if you buy it from here, I get a little bit of commission. It won’t hurt my feelings if you buy it elsewhere; honestly, I’d rather you check it out from your local library, or go to a local book store. I use Bookshop affiliate links instead of Amazon because they distribute a significant chunk of their profits to small, local book stores. ↩

Nick Bostrom
This was, simultaneously, one of the driest and most terrifying books I have ever read.
Really, the conclusion summarized it well:

“Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound.”

It is what the title says: a list of ways we can achieve superintelligence (including, I’d note, a discussion of the fact that it’s both necessary and inevitable), a harrowing discussion of exactly how many ways it can go wrong, and some things we can start trying to do to keep it from going all Skynet on us. Or, as is more likely, wiping out humanity without really noticing, because we were a convenient source of raw materials.
Like I said: terrifying.
But valuable. I’m also convinced this book should be required reading for any AI course. And, y’know, a good chunk of the population beyond that: I count AI as one of the three most likely existential threats out there.¹
So hey, want to somehow be a little bored and scared out of your mind at the same time? Read it.

I’ve got it tied with “Global War, Nuclear” and “Climate Change.” Lower on the list are “A Pandemic With 100% Transmission Rate and 90-Plus Percent Lethality” and “Something From Space.” ↩