Book Review: Superintelligence by Nick Bostrom

Superintelligence: Paths, Dangers, Strategies [Goodreads] by Nick Bostrom is a big idea book. The big idea is that the development of truly intelligent artificial intelligence is the most important issue that our generation will face. According to Bostrom, it may be the most important issue the human race has ever faced. This view suggests that how we approach the development and response to AI will be more important than how we respond to nuclear proliferation, climate change, continued warfare, sustainable agriculture, water management, and healthcare. That is a strong claim.

The sale of Bostrom's book has no doubt been helped by recent public comments by super entrepreneur Elon Musk and physicist Stephen Hawking. Musk, with conviction if not erudition, said

With artificial intelligence we are summoning the demon.  In all those stories where there’s the guy with the pentagram and the holy water, it’s like yeah he’s sure he can control the demon. Didn’t work out.

One almost wishes that Musk didn't live in California. He provided ten million US dollars to the Future of Life Institute to study the issue three months later. Bostrom is on the scientific advisory board of that body.

Hawking agrees with Musk and Bostrom, although without the B movie references, saying,

Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.

Bostrom, Musk and Hawking make some interesting, and probably unfounded, presumptions. This is hardly uncommon in the current public conversation around strong AI. All seem to presume that we are building one or more intelligent machines, that these machines will probably evolve to be generally intelligent, that their own ideas for how to survive will radically differ from ours, and that they will be capable of self-evolution and self-reproduction. 

Jeff Hawkins provides the best answer to Elon Musk, Stephen Hawking, and Nick Bostrom that I have read to date:

Building intelligent machine is not the same as building self-replicating machines. There is no logical connection whatsoever between them. Neither brains nor computers can directory self-replicate, and brainlike memory systems will be no different. While one of the strengths of intelligent machines will be our ability to mass-produce them, that's a world apart from self-replication in the manner of bacteria and viruses. Self-replication does not require intelligence, and intelligence does not require self-replication. (On Intelligence [Goodreads], pp. 215)

Should we not clearly separate our concerns before we monger fear? The hidden presumptions of self-evolution and self-reproduction seem to be entirely within our control. Bostrom makes no mention of these presumptions, nor does he address their ramifications.

At least Bostrom is careful in his preface to admit his own ignorance, like any good academic. He seems honest in his self assessment:

Many of the points made in this book are probably wrong. It is also likely that there are considerations of critical importance that I fail to take into account, thereby invalidating some or all of my conclusions.

Beautifully, a footnote at the end of the first sentence reads, "I don't know which ones." It would be nice to see Fox News adopt such a strategy.

Another unstated presumption is that we are building individual machines based on models of our communal species. Humans may think of themselves as individuals, but we could not survive without each other, nor would there be much point in doing so.

We have not even begun to think about how this presumption will affect the machines we build. It is only in aggregate that we humans make our civilization. Some people are insane, or damaged, or dysfunctional, or badly deluded. Why should we not suspect that a machine built on the human model could not, indeed, would not, run the same risk? We should admit the possibility of our creating an intelligent machine that is delusional in the same way that we should admit the mass delusions of our religious brethren.

Is my supposition too harsh? Consider the case of Ohio bartender Michael Hoyt. Hoyt is not known to have had any birth defects, nor to have suffered physical injury. Yet he lost his job, and was eventually arrested by the FBI, after threatening the life of Speaker of the House John Boehner. Hoyt reportedly heard voices that told him Boehner was evil, or the Devil, or both. He suspected the Boehner was responsible for the Ebola outbreak in West Africa. He told police that he was Jesus Christ. Is Hoyt physically ill, or simply the victim of inappropriate associations in his cortex? We have many reasons to suspect the latter.

Bostrom originally spelled his name with an umlaut (Boström), as befits his Swedish heritage. He apparently dropped it at the same time as he started calling himself "Nick" in place of his birth name Niklas. Bostrom lives in the UK and is now a philosopher at St. Cross College, University of Oxford. Perhaps the Anglicization of his name is as much related to his physical location as the difficulty in convincing publishers and, until recently, the Internet Domain Name System, to consistently handle umlauts. His Web site at nickbostrom.com uses simple ASCII characters.

According to Bostrom, we have one advantage over the coming superintelligence. It is a bit unclear what that advantage is. The book's back jacket insists that "we get to make the first move." Bostrom's preface tells us that "we get to build the stuff." I tend to trust Bostrom's own words here over the publicist's, but think that both are valid perspectives. We have multiple advantages after all.

Another advantage is that we get to choose whether to combine the two orthogonal bits of functionality mentioned earlier, self-evolution and self-replication, with general intelligence. Just what the motivation would be for anyone to do so has yet to be explained by anyone. Bostrom makes weak noises about the defense community building robotic soldiers, or related weapons systems. He does not suggest that those goals would necessarily include self-evolution nor self-replication.

The publisher also informs us on the jacket that "the writing is so lucid that it somehow makes it all seem easy." Bostrom, again in his preface, disagrees. He says, "I have tried to make it an easy book to read, but I don't think I have quite succeeded." It is not a difficult read for a graduate in philosophy, but the general reader will occasionally wish a dictionary and Web browser close at hand. Bostrom's end notes do not include his supporting mathematics, but do helpfully point to academic journal articles that do. Of course, philosophic math is more useful to ensure that one understands an argument being made than in actually proving it.

Perhaps surprisingly, Bostrom makes scant mention of Isaac Asimov's famous Three Laws of Robotics, notionally designed to protect humanity from strong AI. This is probably because professional philosophers have known for some time that they are woefully insufficient. Bostrom notes that Asimov, a biochemistry professor during his long writing career, probably "formulated the laws in the first place precisely so that they would fail in interesting ways, providing fertile plot complications for his stories." (pp. 139)

To be utterly picayune, the book includes some evidence of poor editing, such as adjoining paragraphs that begin with the same sentence, and sloppy word order. I would have expected Oxford University Press to catch a bit more than they did.

Bostrom, perhaps at the insistence of his editors, pulled many philosophical asides into clearly delineated boxes that are printed with a darker background. Light readers can easily give them a miss. Those who are comfortable with the persnickety style of the professional philosopher will find them interesting.

Bostrom does manage to pet one of my particular peeves when he suggests in one such box that, "we could write a piece of code that would function as a detector that would look at the world model in our developing AI and designate the representational elements that correspond to the presence of a superintelligence... If we could create such a detector, we could then use it to define our AI's final values." The problem is that Bostrom doesn't understand the nature of complex code in general, nor the specific forms of AI code that might lead to a general intelligence.

There are already several forms of artificial intelligence where we simply do not understand how they work. We can train a neural network, but we cannot typically deconstruct the resulting weighted algorithm to figure out how a complex recognition task is performed. So-called "deep learning", which generally just means neural networks of more than historical complexity due to the application of more computing power, just exacerbates the problem of understanding. Ask a Google engineer exactly how their program recognizes a face, or a road, or a cat, and they will have no idea. This is equally true in Numenta's Cortical Learning Algorithm (CLA), and will be true of any eventual model of the human brain. Frankly, it is even true of any large software program that has entered maintenance failure, which is almost always an admission by a development team that the program has become too complex for them to reliably change. Bostrom's conception of software is at least as old as the Apple Newton. That is not a complement.

We will surely have less control over any form of future artificial intelligence than it will require to implement his proposed solution. Any solution will not be as simple as inserting a bit of code into a traditional procedural program.

Critically, Bostrom confuses the output of an AI system with its intelligence (pp. 200). This equivalence has been a persistent failure of philosophy. To quote Jeff Hawkins again, who I think sees this particularly clearly,

But intelligence is not just a matter of acting or behaving intelligently. Behavior is a manifestation of intelligence, but not the central characteristic or primary definition of being intelligent. A moment's reflection proves this: You can be intelligent just lying in the dark, thinking and understanding. Ignoring what goes on in your head and focusing instead on behavior has been a large impediment to understanding intelligence and building intelligent machines.

How will we know when a machine becomes intelligent? Alan Turing famously proposed the imitation game, now known as the Turing test, which suggested that we could only know by asking it and observing its behavior. Perhaps we can only know if it tells us without being programmed to do so. Philosophers like Bostrom will, no doubt, argue about this for a long time, in the same way they now argue whether humans are really intelligent. Whatever "really" means.

Bostrom's concluding chapter, "Crunch time", opens with a discussion of the top mathematics prize, the Fields Medal. Bostrom quotes a colleague who likes to say that a Fields Medal indicates that a recipient "was capable of accomplishing something important, and that he didn't." This trite (and insulting) conclusion is the basis for a classic philosophical ramble on whether our hypothetical mathematician actually invented something or whether he "merely" discovered something, and whether the discovery would eventually be made later by someone else. Bostrom makes an efficiency argument: A discovery speeds progress but does not define it. Why he saves this particular argument for his terminal chapter would be a mystery if he had something important to say about what we might do. Instead, he simply tells us to get on studying the problem.

I find that professional philosophers often slip in scale in this way. One moment they are discussing the capabilities and accomplishments of an individual human, generally assumed to be male, and the next they switch to a bird's eye view of our species as if the switch in perspective were justified mid-course. I find this both confusing and disingenuous. It is as if the philosopher cannot bear to view our species from the distance that might yield a more objective understanding.

The actions of individuals, both male and female, are inextricably linked to our cognitive biases. We do not make rational decisions, we make emotional ones, even when we try not to. We make decisions that keep our in-groups stable, by and large. A few, a very few, spend their days trying to think rationally, or exploring the ramifications of rational laws on our near future. A few dare to challenge conventional thinking aimed at in-group stability. Those few are not better than the rest. They are just an outward-looking minority evolved for the group's longer term survival. But the aggregate of our individual decisions looks much like a search algorithm. We explore physical spaces, new foods to eat, new places to be, new ways to raise families, new ways to defeat our enemies. Some work and some don't. Evolution is also a search algorithm, although a much slower one. Our species is where it is because our intelligence has explored more of our space faster and to greater effect. That is both our benefit and our challenge.

The strengths and weaknesses of the professional philosopher's toolbox are just not important to Bostrom's argument. Superintelligence would have been a stronger book if he has transcended them. Instead, it is a litany of just how far philosophy alone can take us, and a definition of where it fails.

I could find no discussion of the various types of approaches to AI, nor how they might play out differently. There are at least five, mostly mutually contradictory, types of AI. They are, in rough historical order:

1. Logical symbol manipulation. This is the sort that has given us proof engines, and various forms of game players. It is also what traditionalists think of when they say "AI".
2. Neural networks. Many problems in computer vision and other sort of pattern recognition problems have been solved this way.
3. Auto-associative memories. This variation on neural networks uses feedback to allow recovery of a pattern when presented with only part of the pattern as input.
4. Statistical, or "machine learning". These techniques use mathematical modeling to solve particular problems such as cleaning fuzzy images.
5. Human brain emulation. Brain emulation may be used to predict future events based on past experiences.

Of these, and the handful of other less common approaches not mentioned, only human brain emulation is currently aiming to create a general artificial intelligence. Not only that, but few AI researchers actually think we are anywhere close to that goal. The popular media has represented a level of maturity that is not currently present.

The recent successes of the artificial intelligence community are a much longer way from general intelligence than one hears from news media, or even some starry eyed AI researchers. There are also good reasons not to worry even if we do manage to create intelligent machines.

Recent news-making successes in AI have been due to the scale of available computing. Examples include the ability for a program to learn to recognize cats in pictures, or to safely drive a car. These successes are impressive, but are wholly specific solutions to very particular problems. Not one of the researchers involved believes that those approaches will lead to a generally intelligent machine. These are tools and nothing but tools. Their output makes us better in the same way that the invention of the hammer or screwdriver, or general purpose computer, made us better. They will not, cannot, take over the world.

Bostrom is, at the end, pessimistic about our chances for survival. Perhaps this is what happens when one spends a lot of time studying global catastrophic risks. Bostrom and Milan M. Cirkovic previously edited a book of essays exploring just such risks in 2011 [Goodreads]. More information is available on the book's Web site. The first chapter is available online. These three paragraphs from Superintelligence anchor his position in relation to AI:

Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound.

For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens.

Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the entire firmament. Nor is there a grown up in sight.

One could imagine the same pessimistic argument being made about nuclear weapons. They must be reigned in before "some little idiot" gets his hands on one. Is that not what has happened? The Treaty on the Non-Proliferation of Nuclear Weapons has been a major force for slowing the spread of nuclear weapons in spite of the five countries that do not adhere to its principles. Separate agreements, threats, and sanctions has so far worked just well enough to plug the holes. Grown ups, from Albert Einstein to the current batch of Strategic Arms Limitation Talks negotiators, have come out of the woodwork when needed. Not only has no one dropped a nuclear bomb since the world came to know of their existence at Hiroshima and Nagasaki, but even the superpowers have willingly returned to small, and relatively low-tech ways of war.

Bostrom urges us to spend time and effort urgently to consider our response to the coming threat. He warns that we may not have the time we think we have. Nowhere does he presume that we will not choose our own destruction. "The universe is change;" said the Roman emperor Marcus Aurelius Antoninus, "our life is what our thoughts make it." Bostrom might learn to temper his pessimism with an understanding of how humans relate to existential threats. Only then do they seem to do the right thing. He might also observe that unexpected events should not be handled using old tools, as noted by the industrialist J. Paul Getty ("In times of rapid change, experience could be your worst enemy.") or management theorist Peter Drucker ("The greatest danger in times of turbulence is not the turbulence; it is to act with yesterday’s logic.") We will need new conceptual tools to handle a new intelligence.

Our erstwhile fear mongers seem also certain that any new general intelligence would, as humans are wont to do, wish to destroy a competing intelligence, us. People fear this not because this is what an artificial intelligence will necessarily be, but because that is what our form of intelligence is. Humans have always feared other humans and for good reason. As historian Ronald Wright noted in A Short History of Progress [Goodreads],

"[P]rehistory, like history, teaches us that the nice folk didn't win, that we are at best the heirs of many ruthless victories and at worst the heirs of genocide."

This raises the fascinating question of how we, as a species, would react to the presence of a newly competitive intelligence on our planet. History shows that we probably killed off the Neanderthals, as earlier human species killed off Homo Erectus and our earlier predecessors. We don't play well with others. Perhaps our own latent fears will insist on the killing off of a new, generally intelligent AI. We should consider this nasty habit of ours before we worry too much about how a hypothetical AI might feel about us. If an AI considers us a threat, should we really blame it? We probably will be a threat to its existence.

It is possible that a single hyper-intelligent machine might not even matter much in the wider course of human affairs. Just like the natural, generational genius does not always matter. The history of the human race seems to be more dominated by the slow, inexorable march of individual decisions than it is by the, often temporary, upheavals of the generational genius. How would human development have changed if the Persian commander Spithridates had succeeded in killing Alexander the Great at the Battle of the Granicus? He almost did. Spithridates' axe bounced off Alexander's armor. Much has been made of the details, but people would still spread through competition, and contact between East and West would still have eventually occurred. The difference between having a genius and not having a genius can be smaller than we think in the long run.

Bostrom's main point is that we should take the development of general artificial intelligence seriously and plan for its eventual regulation. That's fine, for what it is worth. It is not worth very much, really. We are much more likely to react once a threat emerges. That's what humanity does. Bostrom is at best early at delivering a warning and at worst barking up the wrong tree.