To put it in a single sentence, I’d say that it’s because only a
minority of cognitively possible goal sets place a high priority on the
continued survival of human beings and the structures we value.
Another reason is that we can’t specify what we value in enough
mathematical detail to transfer it to a new species without a lot of
requisite hassle.
It would be easy if we could just transfer over the goal set of a
“typical human” or a “nice person” and hope for the best. But there’s a
problem: we have no experimental evidence of what happens when a human
being can modify its own goals, or increase its own intelligence and/or
physical power exponentially.
What little evidence we have of scenarios where people acquire a lot
of power in a short amount of time indicates that the outcomes are
usually not pretty. In fact, we have complicated democratic mechanisms
built into our society to guard against these types of outcomes.
Most AI designers are missing the challenge because no one wants to
have to take the responsibility of creating the first truly intelligent
being. They just want to play with their program. The idea of taking
any responsibility for the products of one’s research is a relatively
recent notion, one that only holds weight with a minority of scientists
and engineers, even today. This is usually because scientists and
engineers are embedded in a large institutional apparatus that places
responsibility so far up the chain of command that the actual
researchers are absolved of most, if not all responsibility.
Back to the original issue of goal sets. Here are some likely
applications for the most advanced AI technologies in the next 10-20
years:
- Intelligence analysis and wargaming. (link)
- Law enforcement (link)
- Analyzing interstate politics (link)
- Finance, banking, & investing (link)
- Controlling combat robots (link)
- Automating work flows (link)
There are many others, but I put these on the top of the list
because they have the most economic or political importance, and
therefore will be getting the most research money.
As AI in these areas progresses, the systems will go from outputting
decisions only when explicitly requested, to outputting decisions
continually and automatically. When a human worker consults the machine
for input, it will be more like dipping a cup into a stream and tapping
into the preexisting flow of knowledge consolidation and
decision-making, rather than flicking on a light switch or pressing
“run” for a conventional computer program.
Being continuously thinking, continuously decision-making entities,
these AI systems will have implicit top goals, whether people
explicitly program them or not. The implicit top goal of a workflow
automator will be to accelerate the completion of productive tasks. The
implicit top goal of the finance bot will be to pick stocks that
maximize return on investment. The implicit top goal of the combat
robot AIs will be to take out or capture people specified by certain
data files in its memory.
What makes AI potentially so dangerous is the lack of background
common sense and humanness that we take for granted. When the clock
hits 5, most workers put down their tasks and are done for the day.
They go home and spend time with their family, watch TV or play games,
or just relax. An artificial worker would have no such “background
normality” unless we program it in. It’s on task, 24 hours a day, 7
days a week, as long as its computer continues to suck power from the
wall.
It’s that kind of monomaniacal devotion that puts humanity at risk
from AI when it begins to step out of the lab and into the real world.
An AI with implicit top goals will want to reinforce those goals and
achieve them more effectively, where the “goals” are not the same as
what you’d see in a human that was handed a piece of paper with those
goals written on it, but as they are represented in the context of the
AI’s decision structure and worldview.
Reasonableness and sensibility about goals are not easy to transfer
over to a mind without the knowledge and common sense built into every
neurologically normal human being. A blank slate intelligence sitting
in the middle of a forest would be able to build models and make
inferences about numerous aspects of its surroundings - that trees are
tall, that animals are mobile but plants aren’t, that the weather
changes in cycles. But inferences about “the right thing to do”? You
can’t derive an ought from an is. Putting an AI in a social environment
with humans or other AIs doesn’t help, because without some deep-seated
motivation to care about this weird “morality” thing in the first
place, an AI will just happily go about accomplishing the
subtlety-devoid goals it was originally assigned. As it gains the
ability to improve on its own intelligence or tap into the power of
robotics, it will continue to get better and better at achieving those
goals and harder and harder for humans to reach in and grant it the
motivation to care about morality in the abstract.
If AIs in any of the applications I listed before gained the ability
to improve upon themselves significantly, either mentally or
physically, the implicit top goals they were given will be magnified
many times over. There would be little reason for the AI to modify
those goals unless such flexibility mechanisms were explicitly
programmed in. When a human sees someone starving, they tend to feel
sorry for them and at the very least wish they could help. When a human
sees someone attacking a defenseless child, they tend to get angry. To
your typical AI, a person starving or a child being attacked is only
relevant in the context of the goals it already has - “how does this
starving human affect stock prices?”, or “can this starving human give
me information regarding the location of my next target?” are two
inquiries that might come to mind.
Freedom, empathy, self-determination, consensus-building, conflict
resolution, aesthetics, camaraderie and rapport - these values and
inclinations are built in automatically for every human without serious
brain defects. For an AI to share them, they have to be put in terms of
lines of code and mathematical rigor. What programmer has the time to
do all that work when general intelligence without the human-like
morality will be significantly easier to achieve?
It’s that difficulty disparity between stripped-down general
intelligence and morally-sophisticated general intelligence that makes
AI so dangerous in the long term.