To put it in a single sentence, I’d say that it’s because only a minority of cognitively possible goal sets place a high priority on the continued survival of human beings and the structures we value.

Another reason is that we can’t specify what we value in enough mathematical detail to transfer it to a new species without a lot of requisite hassle.

It would be easy if we could just transfer over the goal set of a “typical human” or a “nice person” and hope for the best. But there’s a problem: we have no experimental evidence of what happens when a human being can modify its own goals, or increase its own intelligence and/or physical power exponentially.

What little evidence we have of scenarios where people acquire a lot of power in a short amount of time indicates that the outcomes are usually not pretty. In fact, we have complicated democratic mechanisms built into our society to guard against these types of outcomes.

Most AI designers are missing the challenge because no one wants to have to take the responsibility of creating the first truly intelligent being. They just want to play with their program. The idea of taking any responsibility for the products of one’s research is a relatively recent notion, one that only holds weight with a minority of scientists and engineers, even today. This is usually because scientists and engineers are embedded in a large institutional apparatus that places responsibility so far up the chain of command that the actual researchers are absolved of most, if not all responsibility.

Back to the original issue of goal sets. Here are some likely applications for the most advanced AI technologies in the next 10-20 years:

• Intelligence analysis and wargaming. (link)
• Law enforcement (link)
• Analyzing interstate politics (link)
• Finance, banking, & investing (link)
• Controlling combat robots (link)
• Automating work flows (link)

There are many others, but I put these on the top of the list because they have the most economic or political importance, and therefore will be getting the most research money.

As AI in these areas progresses, the systems will go from outputting decisions only when explicitly requested, to outputting decisions continually and automatically. When a human worker consults the machine for input, it will be more like dipping a cup into a stream and tapping into the preexisting flow of knowledge consolidation and decision-making, rather than flicking on a light switch or pressing “run” for a conventional computer program.

Being continuously thinking, continuously decision-making entities, these AI systems will have implicit top goals, whether people explicitly program them or not. The implicit top goal of a workflow automator will be to accelerate the completion of productive tasks. The implicit top goal of the finance bot will be to pick stocks that maximize return on investment. The implicit top goal of the combat robot AIs will be to take out or capture people specified by certain data files in its memory.

What makes AI potentially so dangerous is the lack of background common sense and humanness that we take for granted. When the clock hits 5, most workers put down their tasks and are done for the day. They go home and spend time with their family, watch TV or play games, or just relax. An artificial worker would have no such “background normality” unless we program it in. It’s on task, 24 hours a day, 7 days a week, as long as its computer continues to suck power from the wall.

It’s that kind of monomaniacal devotion that puts humanity at risk from AI when it begins to step out of the lab and into the real world. An AI with implicit top goals will want to reinforce those goals and achieve them more effectively, where the “goals” are not the same as what you’d see in a human that was handed a piece of paper with those goals written on it, but as they are represented in the context of the AI’s decision structure and worldview.

Reasonableness and sensibility about goals are not easy to transfer over to a mind without the knowledge and common sense built into every neurologically normal human being. A blank slate intelligence sitting in the middle of a forest would be able to build models and make inferences about numerous aspects of its surroundings - that trees are tall, that animals are mobile but plants aren’t, that the weather changes in cycles. But inferences about “the right thing to do”? You can’t derive an ought from an is. Putting an AI in a social environment with humans or other AIs doesn’t help, because without some deep-seated motivation to care about this weird “morality” thing in the first place, an AI will just happily go about accomplishing the subtlety-devoid goals it was originally assigned. As it gains the ability to improve on its own intelligence or tap into the power of robotics, it will continue to get better and better at achieving those goals and harder and harder for humans to reach in and grant it the motivation to care about morality in the abstract.

If AIs in any of the applications I listed before gained the ability to improve upon themselves significantly, either mentally or physically, the implicit top goals they were given will be magnified many times over. There would be little reason for the AI to modify those goals unless such flexibility mechanisms were explicitly programmed in. When a human sees someone starving, they tend to feel sorry for them and at the very least wish they could help. When a human sees someone attacking a defenseless child, they tend to get angry. To your typical AI, a person starving or a child being attacked is only relevant in the context of the goals it already has - “how does this starving human affect stock prices?”, or “can this starving human give me information regarding the location of my next target?” are two inquiries that might come to mind.

Freedom, empathy, self-determination, consensus-building, conflict resolution, aesthetics, camaraderie and rapport - these values and inclinations are built in automatically for every human without serious brain defects. For an AI to share them, they have to be put in terms of lines of code and mathematical rigor. What programmer has the time to do all that work when general intelligence without the human-like morality will be significantly easier to achieve?

It’s that difficulty disparity between stripped-down general intelligence and morally-sophisticated general intelligence that makes AI so dangerous in the long term.

Humans find it hard to imagine intelligences smarter than we are because we’re designed by evolution to ignore the problems we can’t solve and focus on those we can. Doing it any other way would be an inappropriate use of resources.

What are the top five elements in your body and their relative proportions? You can’t answer? What’s taking you so long? You don’t even know what you’re made of?

Fact is, humans are pretty damn stupid. Not stupid relative to me or stupid relative to Einstein, but stupid in the scheme of things. Stupid relative to what we could be. We can offer any number of excuses, but in the end they’re nothing but excuses.

Homo sapiens evolved out of the primordial muck. We’re what happens when the muck gets just barely smart enough to reflect upon itself and manipulate its environment significantly.

There are two anthropic pressures at play here. Let’s assume, like Max Tegmark and other physicists, that we live in a gigantic multiverse where all possibilities are realized. The sector of the multiverse capable of harboring intelligent life, or life of any type, is extremely small. If our spatial dimensionality were different, or the intensity of the strong force, or the fine structure constant, or any number of other fundamental constants varied by even a tiny bit, life in this universe would be impossible. Tipler and Barrow beat this point into the ground in The Anthropic Cosmological Principle, but we’ve seen it already from numerous physicists.

The first anthropic pressure is the probabilistic bias towards chaos, disorder, and inhospitability to life, intelligent life in particular. In most of the multiverse life is impossible. But in some tiny portion, in which we (surprise!) happen to find ourselves, intelligent life just barely was able to evolve out of the muck and acquire enough cognitive complexity to consciously kill each other and compete for mates instead of just doing so mindlessly.

The second anthropic pressure is slightly more speculative. It’s the idea that intelligent species that are too smart wipe themselves out too quickly to really get anywhere. They build self-improving AIs that ignore their creators and tile the cosmic neighborhood with value structures that are a mere shadow of what the programmers originally meant, or launch superintelligent uploads who slowly, and then quickly become obsessed with the idea of constantly stimulating their own pleasure centers to the exclusion of all other pursuits. Both outcomes radically reduce the number of conscious individuals in existence after that point, thereby selecting those quadrants of spacetime out of the anthropic lottery. We’re unlikely to be born into those regions because they are relatively uninhabited, just like we’re unlikely to be born in universes where infant stars have so much gravity that their accretion discs get sucked in before forming stable planets.

We are born in regions that are typical. Industrial civilizations filled with billions of non self-modifying intelligent social animals, apparently. We’re relatively unintelligent because 1) we just evolved from the muck and 2) because we haven’t been clever enough to destroy ourselves yet. Two factors, any one of which alone would be enough to hold the argument up.

But, worry not. There is no reason to despair. These anthropic arguments for our relative stupidity only underscore our potential for growth. We can improve our quality of life to new heights we could never even dream of.

There is an issue of concern, however. If the future is so much more prosperous and populous than today, then why don’t we find ourselves there, instead of here? If out of every 1,000,000 random beings, only one finds itself in civilizations with only a few billion people, then is it just an enormous coincidence that we happen to find ourselves here?

Coincidence is not good enough. There are reasons to believe that this probabilistic issue is a huge problem. It’s called the Doomsday Argument. You can find numerous rebuttals in the Wikipedia article, but many of them are quite subtle, and if you dismiss the argument merely based on its implications, then I think we can justifiably throw out your opinion.

What is your reason for dismissing the Doomsday Argument? Or if you don’t have one, how do you cope?

Cross-posted from Accelerating Future: 

An 'ethical robotics' bit has started hitting the AI news sites today, this time from tech philosopher Bernhard Irrgang at the Dresden University of Technology.  As always, I ask, "does this person have a clue?"  And from reading his paper, "Ethical Acts in Robotics", the answer is, unfortunately, a firm no.

From the opening:

More:

Opaque, italicized and capitalized philosophical dribble-words, check.  Reviews of Descartes' thinking as if we had never heard of it before reading this paper, check.  Conception of emotions as a detached mindstuff-object supervening on normal cognition rather than an inevitable product of incremental evolution from pre-intelligent descendents, check.  Repudiation of posthumanism in the conclusion, check.

Why is it so hard to make concrete progress on the problem of how to write an intelligent software program that behaves such that we don't regret creating it?  A lack of experimental evidence is certainly a contributing factor, but theoretical progress is certainly possible, and we already know of dozens of immediate roadblocks that screw peoples' ideas up before they even get started.  Blatant anthropocentrism, unfamilarity with cognitive science, confusing metaphorical statements for ontological statements, mistaking words and phrases for the structure of cognitive content, etc.  I could list dozens of common errors that get in the way when we ask the simple question, "how would we build a machine that acts a certain way?"

Over the past six years, I've heard hundreds of proposals for approaches to building intelligent machines that are on humanity's side.  Sadly, it remains the case that the only proposals that have any real value are those coming out of the Singularity Institute for Artificial Intelligence.  The problem is certainly not solved, and any solution will likely require the efforts and input of many more geniuses than the number that are currently working at SIAI.  

What is Friendly AI?  There is a good place to start asking this question, and it can be found on the Singularity Institute's website.  However, to make matters simpler, I will post it here.  The tagline of this short list of 24 definitions is "looking at the issue of AI morality from enough different angles to at least try and show what the question is":

• Friendly AI is the art of designing and constructing minds that play a positive role in the universe.
• A "Friendly AI" is an AI that takes actions that are, on the whole, beneficial to humans, humanity, and sentient life; actions that are benevolent rather than malevolent, nice rather than hostile.
• Friendly AI is the means by which we ensure that transhuman intelligence in AIs is linked to transhuman morals, altruism, wisdom, and philosophy.
• A Friendly seed AI is one such that the resulting Singularity is at least as good as a Singularity sparked by any individual human or combination of humans.
• Friendliness is a "metawish" - a way of saying to an AI:  "When you grow up, grow into what we would have made you to be, if we were as smart as you."
• A Friendly AI is an AI that wants to be Friendly - one which, like Martin Luther King or Gandhi, is altruistic not because someone is forcing it to be altruistic but because that is the AI's own choice. Gandhi had the power to stop being altruistic at any time, but chose not to do so.
• Friendly AI is an attempt to get rid of the concept of "Asimov Laws" (a science-fictional plot device invented in the 1940s) and replace it with a serious discipline.
• Friendly AI is an attempt to get rid of the concept of "Asimov Laws" (external programmatic constraints on an AI) and replace it with a solution which works even if the AI has unrestricted access to its own source code.
• Friendly AI is an attempt to get rid of the concept of "Asimov Laws" (coercive restrictions placed on AIs, by humans, for essentially selfish purposes) and get past the "us vs. them" attitude that currently permeates discussion of AI.
• Friendly AI is the set of technical and moral issues involved with standing, not just in loco parentis, but in loco evolution, to a new intelligent species.
• Friendly AI is a channel for transferring human morality which obeys the constraint that each statement communicated, on any matter of fact or morality, is honestly believed or personally held by the programmer(s) making the statement.
• Friendly AI is the art of constructing an AI morality such that the final result is not sensitive to the choice of which particular programmers built the AI, as long as the programmers had the basic belief that an AI should try to avoid sensitive dependency on the choice of initial programmer.
• Friendly AI is a strategy for ensuring that the personal quirks and philosophical errors of the original programmers don't remain fixed in the mind of the AI after it grows up.
• A Friendly AI is a mind that, as it grows, grows into whatever a human upload would grow into as intelligence approached infinity as a limit. (If it makes a difference, this can be further qualified by specifying that the human upload starts out with a philosophical commitment to rationality and altruism, or even that the human upload starts out as a pure altruist.)
• Friendly AI is a strategy for ensuring that the Singularity transcends the quirks and errors of whichever technological civilization first creates self-improving minds (i.e: only two generations ago in the United States, blacks rode in the back of the bus; we don't know what deep and shallow mistakes may persist in our contemporary conceptions of morality).
• A Friendly AI is an AI that is "on the side" of sentient life.
• Friendly AI is the technological pathway required to defend the integrity of the class of Singularities originally sparked by an AI.
• A Friendly AI is a human-equivalent philosopher.
• Friendly AI is the strategy by which the basic challenge of constructing an AI morality is transferred over to the AI itself, so that the problem can be handled or reconsidered by the transhuman intelligence of an AI that surpasses human intelligence.
• Friendly AI is a way of transferring to a transhuman AI the question "What morality should be given to a transhuman AI?"
• Friendly AI is an attempt to create an AI, which, when it grows into a transhuman, will be capable of dealing with issues that exhibit dependency on the philosophical question:  "What is good, what is evil, and how should we be asking this question?"
• Friendly AI is the attempt to describe the complex functional adaptations underlying metamorality - the forces that influence how humans think when faced with a choice between alternative moral or philosophical systems.
• Friendly AI is the attempt to describe the structure of human cognition about metamorality in sufficient detail to include all structural properties underlying our intuitive understanding of the metawish "Be the best AI we could have constructed you to be."
• Friendly AI is the attempt to construct a nonanthropocentric theory of cognitive processes underlying moral and metamoral reasoning, with sufficient generality to predict the differential results of different AI designs.

For more on Friendly AI, see 'Beyond anthropocentrism'.