Famed science-fiction writer Fredric Brown (1906–1972) delighted in creating the shortest of short stories. “Answer,” published in 1954, encapsulated a prescient meditation on the future of human-machine relations within a single double-spaced, typewritten page.

The foreboding of the story echoes current apprehensions of scientists, policy makers and ethicists over the rapid evolution of machine intelligence.

“Answer” begins under the watchful eyes of a dozen television cameras that are recording the ceremonial soldering of the final connection to tie together all the “monster” computers in the universe.

The machines are about to link 96 billion planets into a single “supercircuit” that combines “all the knowledge of all the galaxies.”

Two witnesses on the scene are identified only as Dwar Ev and Dwar Reyn. After throwing the switch that connects the galactic circuit, Dwar Ev suggests to his companion that he ask the machine the first question:

“Thank you,” said Dwar Reyn. “It shall be a question which no single cyber netics machine has been able to answer.”

He turned to face the machine. “Is there a God?”

The mighty voice answered without hesitation, without the clicking of a single relay.

“Yes, now there is a God.”

Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.

A bolt of lightning from the cloudless sky struck him down and fused the switch shut.

We are in the midst of a revolution in machine intelligence, the art and engineering practices that let computers perform tasks that, until recently, could be done only by people. There is now software that identifies faces at border crossings and matches them against passports or that labels people and objects in photographs posted to social media. Algorithms can teach themselves to play Atari video games. A camera and chip embedded in top-of-the-line sedans let the vehicles drive autonomously on the open road.

What separates these agents from earlier success stories, such as IBM's Deep Blue, which beat the world's reigning chess champion in 1997, and IBM's Watson, which accomplished the same for the quiz show Jeopardy in 2011, is that they are taught by trial and error. The new wave of artificial intelligence (AI) is based on insights derived from the way animals and people learn and analysis of the underlying brain circuits that allowed theorists to develop supervised learning algorithms: the software is shown an image, and depending on whether or not it correctly identifies the face or increases the video game score, parameters internal to the program (so-called synaptic weights) are minutely adjusted. Such machine learning, if done over trillions of machine cycles (yes, it is very computing-intensive), can lead to systems that match or, in some cases, exceed human performance metrics. And, of course, the algorithm never gets distracted or tired and remains focused, day and night (see my July/August column “Intelligence without Sentience”).

Within a decade these instances of “weak” or “narrow” AI—able to replicate specific human tasks—will permeate society. Siri is only the beginning. Driverless cars and trucks will become the norm, and our interactions in supermarkets, hospitals, industry, offices and financial markets will be dominated by narrow AI. The torrid pace of these advances will put severe stress on society to deal peacefully with the attendant problems of unemployment (the U.S. trucking industry alone employs several million drivers) and growing inequality.

Obscured by this razzle-dazzle progress is how far away we remain from “strong” or “general” AI, comparable to the intelligence of the proverbial man or woman in the street who can navigate a car, hurtle on skis down a mountain slope, carry on a conversation about pretty much any topic—often in two or more languages. That same ordinary individual might also play a variety of games, serve on a jury and plan for retirement decades in the future. Hampering our ability to design general AI is the embarrassing fact that we don't understand what we mean by “intelligence.” This lack of knowledge makes any predictions of when we will achieve strong AI fraught with uncertainty. Still, it may not be so far away. For the record, most experts believe that strong machine intelligence will arrive before the century is over, assuming current trends continue.

Superintelligence: Paths, Dangers, Strategies deals with the aftermath of that event. The book's author, Nick Bostrom, a professor of philosophy at the University of Oxford, has a background in theoretical physics and neuroscience. His scholarly work is focused on understanding and mitigating emerging risks that threaten the very survival of the human species: full-blown nuclear warfare, massive climate change, synthetic biology, nanotechnology or runaway machine intelligence. Superintelligence deals with the last. I warmly recommend the opening and the closing chapters for their enticing arguments, soaring metaphors and insightful fables. You will come away unsettled, if not downright frightened.

The distribution of human intelligence across any representative population is bell-shaped, with the feebleminded at one end and the geniuses at the other. But there is no natural law that stipulates that humans as a group are as intelligent as they could be in an ideal world. Indeed, Homo sapiens is plagued by superstitions and short-term thinking (just watch politicians, many drawn from our elites, to whom we entrust our long-term future). To state the obvious, humanity's ability to calmly reason—its capacity to plan and build unperturbed by emotion (in short, our intelligence)—can improve. Indeed, it is entirely possible that over the past century, average intelligence has increased somewhat, with improved access to good nutrition and stimulating environments early in childhood, when the brain is maturing.

And what is true of the biological variety should also be true of its artificial counterpart. There is no discernible principle that would prevent emergence of an AI that is more intelligent than the average person or even any person alive. Indeed, given the competition among the various organizations capable of designing AI systems—mainly national governments and private corporations—their engineers will design ever smarter machines that outperform opponents, whether human or cyborg, and maximize their own gain. This is likely to involve the ability of machines to self-improve by trial and error and by reprogramming their own code. What might happen when machines start to boost their own intelligence was first pointed out by mathematician Irving John Good in a memorable passage in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind…. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

Bostrom considers different forms of superintelligence: qualitative ones—say, Albert Einstein versus someone intellectually challenged; collective ones, a team of Einstein-level geniuses; or quantitative ones, such as an intelligence that invents the theory of general relativity within an hour of first thinking about the fundamental nature of spacetime rather than the decade that it took Einstein to develop the theory. For Bostrom's reckoning of existential risks, it doesn't much matter as long as the AI can outthink people. And there might be no warning that the age of machines has arrived, nothing like the sonic boom first heard above California's skies in 1947, when the X-1 plane broke the sound barrier, to herald the birth of a superintelligent AI.

Bostrom's book does not explain how this stupendous task could be accomplished; his is not a guide on how to program a strong AI machine to have flexible goals, understand speech and engage in long-term planning. Rather invoking nothing but the iron laws of physics and mathematical logic, the bulk of his thesis is an extended lucubration on the many evolutionary trajectories a superintelligence could take: Will there be many AIs, or will a single malevolent one emerge at a planetary scale? What will an all-consuming machine intelligence try to do—to us, to the planet? How will we control it? Will we even be able to?

Bostrom seeks to hash out the implications of an emergent AI and ways to erect safeguards against the threatening outcomes that are the tropes of science-fiction movies and in stories such as Brown's “Answer.” The potential dangers posed by such a machine do not depend on how smart it is but on what its ultimate goals are. Indeed, an AI doesn't even have to be supersmart to be a grave threat to humanity—a narrow AI designed to maximize “return on investments” at all costs in its calculations could trigger a war or some other calamity and thereby rake in untold billions by hedging stocks in the affected industries. Or a narrow military AI connected to our network of nuclear-tipped missiles could unleash a devastating preemptive first strike on the principle that waiting longer would maximize the number of its own citizens dying in nuclear hellfire.

What concerns Bostrom is the unpredictability of what might happen when the technology starts edging toward acquiring the capabilities of a strong AI that takes its goals to extremes never intended by its original programmers. A benign superintelligence that wants nothing but happy people might implant electrodes into the brain's pleasure centers, to deliver jolts of pure, orgasmic gratification. Do we really want to end up as wire-heads? And what about the innocent paper-clip-maximizing AI that turns the entire planet and everything on its surface into gigantic, paper-clip-making factories? Oops.

Given humanity's own uncertainty about its final goals—being as happy as possible? Fulfilling the dictum of some holy book so we end up in heaven? Sitting on a mountaintop and humming “Om” through nostrils while being mindful? Colonizing the Milky Way galaxy?—we want to move very deliberately here.

Things turn out to be no easier when considering how to control such entities. The best known rules to constrain their behavior do not come from roboticists or philosophers but from science-fiction author and biochemist Isaac Asimov. The first of his three laws of robotics (conceived more than 70 years ago!) states: “A robot may not injure a human being or, through inaction, allow a human being to come to harm.”

Although this appears reasonable, it is utterly inadequate for dealing with life's messiness. Armed forces have to be ready to quickly and effectively incapacitate a large number of opposing soldiers to prevent a greater evil from coming to pass. Should a superintelligence therefore forestall all armed conflict? Should this AI shut down pollution-producing industries to counter global warming at the cost of a decade-long worldwide depression? Does the first law apply to unborn fetuses and to patients in coma?

Bostrom is most concerned with what he calls the “control problem,” the challenge of how to engineer superintelligent machines so as to achieve outcomes that are safe and beneficial for humanity. This goal cannot be achieved by simply picking a set of ethical rules and implementing these into specific instructions. Traditionally the job of the political systems and the courts is to enforce such written laws and the unwritten code that governs society. These objectives are often in conflict with each other: the powerful “thou shalt not kill” edict is routinely violated on the battlefield, on death row, in terminating pregnancies and in slaughterhouses.

Of course, as Bostrom caustically remarks, humankind can hardly claim to be basking in the high noon of perfect moral enlightenment. People can't seem to agree on the best rules to live by. Should an ascendant AI follow the U.S. Constitution, rules laid down by the Chinese Communist Party or dictates of the mullahs in Iran?

The full gamut of possibilities for how an intelligence might behave is simply too vast to be constrained in any meaningful manner by what can't be ruled out by physics. Many options are extremely unlikely. For example, Bostrom goes off on a tangent about the possibility that an AI system believes it exists in an entirely simulated universe. Or he assumes that any superintelligence worthy of its name could eliminate the risks from asteroid impacts or natural pandemics and would also spread itself throughout the entire universe. To assume all of this as a given seems absurd.

But his basic theory should be taken seriously. To constrain what could happen and ensure that humanity retains some modicum of control, we need to better understand the only known form of intelligence. That is, we need to develop a science of intelligence by studying people and their brains to try to deduce what might be the ultimate capabilities and goals of a machine intelligence. What makes a person smart, able to deal with a complex world that is in constant flux? How does intelligence develop throughout infancy, childhood and adolescence? How did intelligence evolve?

How much does intelligence depend on being embedded in social groups? What is the relation between intelligence and emotion and between intelligence and motivation? And what about consciousness? Will it make a difference to the AI's action if it feels something, anything, and if it, too, can experience the sights and sounds of the universe?

In a field largely defined by science-fiction novels and movies acting as laboratories for exploring the possible, Bostrom's Superintelligence is a philosopher's Cassandra call to action (adorned with more than 40 pages of endnotes). Woe to us if we don't eventually tackle the questions that the book throws out. Doing so effectively will be possible only once we have a principled, scientific account of the internal constraints and the architecture of biological intelligence. Only then will we be in a better position to put effective control structures in place to maximize the vast benefits that may come about if we develop smart companions to help solve the myriad problems humankind faces.