Did HAL Commit Murder?
By Daniel C. Dennett Rough draft for Stork Volume, February 28, 1996 The first robot homicide
was committed in 1981, according to my files. I have a yellowed clipping
dated 12/9/81 from the Philadelphia Inquirer--not the National
Enquirer--with the headline: Robot
killed repairman, The story was an anti-climax: at
the Kawasaki Heavy Industries plant in to have performed a legally
prohibited action, such as killing another human being; one must have done so
with a culpable state of mind, or mens rea. Such culpable mental
states are of three kinds: they are either motivational states of purpose,
cognitive states of belief, or the nonmental state of negligence. [Cambridge
Dictionary of Philosophy 1995, p.482] The legal concept has no
requirement that the agent be capable of feeling guilt or remorse or
any other emotion; "cold-blooded" murderers are not in the
slightest degree exculpated by their flat affective state. Star Trek's Spock
would fully satisfy the mens rea requirement in spite of his fabled
lack of emotions. Drab, colorless--but oh so effective--"motivational
states of purpose" and "cognitive states of belief" are enough
to get the fictional Spock through the day quite handily, and they are well
established features of many existing computer programs.
Deep Blue, like many
other computers equipped with AI programs, is what I call an intentional
system: its behavior is predictable and explainable by attributing to it
beliefs and desires--"cognitive states" and "motivational
states"--and the rationality required to figure out what it ought to do
in the light of those beliefs and desires (Dennett, 1971, 1987) Are these
skeletal versions of human beliefs and desires sufficient to meet the mens
rea requirement of legal culpability? Not quite, but it is hard to see
what is missing if we restrict our gaze to the limited world of the chess
board. Since cheating is literally unthinkable to a chess playing computer
such as Deep Blue, and since there are really no other culpable actions
available to an agent restricted to playing chess, there are no ready
examples of cases where we might want to blame a chess playing computer for
anything, let alone convict it of a crime. But we also assign responsibility
to agents in order to praise or honor the appropriate agent. Who or what
deserves the credit for beating Kasparov? Deep Blue is clearly the best
candidate. Yes, we may join in congratulating Feng-hsiung Hsu and the IBM
team on the success of their handiwork, but in the same spirit we might
congratulate Kasparov's teachers, handlers, and even his parents. But no
matter how assiduously they may have trained him, drumming into his head the
importance of one strategic principle or another, they didn't beat
Deep Blue in the series; Kasparov did. Deep Blue is the best
candidate for the role of responsible opponent of Kasparov, but this is not
good enough, surely, for full moral responsibility. If we expanded Deep
Blue's horizons somewhat, it could move out into the arenas of injury and
benefit that we human beings operate in. It's not hard to imagine a touching
scenario in which a grand master deliberately (but oh so subtly) throws a
game to an opponent, in order to save a life, or to avoid humiliating a loved
one, or to keep a promise, or . . . . (make up your own O'Henry story here).
Failure to rise to such an occasion might well be grounds for blaming a human
chess player. Winning or throwing a chess match might even amount to the
commission of a heinous crime (make up your own Agatha Christie story here).
Could Deep Blue's horizons be so widened? (For more on this theme, see
Haugeland 1993) Deep Blue is an
intentional system, with beliefs and desires about its activities and
predicaments on the chess board, but in order to expand its horizons to the wider
world of which chess is a relatively trivial part, it would have to be given
vastly richer sources of "perceptual" input--and the means of
coping with this barrage in real time. Time pressure is of course already a
familiar feature of Deep Blue's world. As it hustles through the
multi-dimensional search tree of chess, it has to "keep one eye" on
the clock, but the problems of optimizing its use of time would increase by
orders of magnitude when it had to juggle all these new concurrent projects
(of simple perception and self-maintenance in the world, to say nothing of
more devious schemes and opportunities). For this hugely expanded task of
resource management, it would need extra layers of control--above and below
its chess playing software. Below, it would need to be "innately"
equipped with a set of rigid traffic control policies embedded in its
underlying operating system, just to keep its perceptuo-locomotor projects in
basic coordination. Above, it would have to be able to pay more attention to
features of its own expanded resources, always on the lookout for inefficient
habits of thought, strange loops (Hofstadter, 1979), obsessive ruts,
oversights, and dead-ends. It would have to become a higher-order
intentional system, in other words, capable of framing beliefs about its own
beliefs, desires about its desires, beliefs about its fears about its
thoughts about its hopes, . . . Higher-order
intentionality is a necessary condition for moral responsibility (Dennett,
1976), and Deep Blue exhibits little sign of such capabilities. There is of
course some self-monitoring implicated in any well-controlled search: Deep
Blue doesn't make the mistake of re-exploring branches it has already
explored, for instance, but this is innate policy designed into the underlying
computational architecture, not anything under flexible control. Deep Blue
can't converse with you--or with itself--about the themes discernible in its
own play; it's not equipped (so far as I know--David, correct me if you
know better; I haven't found a good source of detailed information on the
structure and capabilities of Deep Blue!) to notice--and analyze,
criticize, analyze, manipulate--the fundamental parameters that determine its
policies of heuristic search or evaluation. Adding the layers of software
that would permit Deep Blue to become self-monitoring and self-critical, and
hence teachable, in all these ways would dwarf the already huge Deep Blue
programming project--and turn Deep Blue into a radically different sort of
agent. HAL purports to be just
such a higher-order intentional system--and he even plays a game of chess
with Dave. HAL is an enhancement of Deep Blue equipped with eyes and ears,
and a large array of sensors and effectors distributed around in Discovery
One, the space ship. A familiar factoid about HAL is the curious observation
that its name is IBM-minus-one--just roll back each letter one place in the
alphabet. According to Arthur C. Clarke's book (p.92), however,
"HAL" stands for "Heuristically programmed Algorithmic
computer," a feature Hal and Deep Blue have in common in any case. HAL
is not at all garrulous or self-absorbed, but in his few speeches he
expresses an interesting variety of higher-order intentional states, from the
most simple to the most devious: "Yes, it's puzzling.
I don't think I've ever seen anything quite like this before." HAL doesn't just respond
to novelty with a novel reaction; he notices that he is encountering novelty,
a feat that requires his memory to have an organization far beyond that
required for simple conditioning to novel stimuli. (See Dennett, 1993, 1996,
for more on the differences between conditioning or "ABC" learning
and its fancier descendants.) "I can't rid myself
of the suspicion that there are some extremely odd things about this
mission." "I never gave these
stories much credence, but particularly in view of some of the other things
that have happened, I find them difficult to put out of my mind." HAL has problems of
resource management not unlike our own. Obtrusive thoughts can get in the way
of other activities. The price you pay for adding layers of flexible
monitoring in order to keep better track of your own mental activities is . .
. more mental activities to keep track of! "I've still got the
greatest enthusiasm and confidence in the mission. I want to help you." Another price you pay for
higher-order intentionality is the opportunity for duplicity, and it
comes in two flavors: self-deception and other-deception. This layering of
the mind is recognized by Friedrich Nietzsche as the key ingredient in making
a moral animal, and in his overheated prose it becomes the
"priestly" form of life: For with the priests everything
becomes more dangerous, not only cures and remedies, but also arrogance,
revenge, acuteness, profligacy, love, lust to rule, virtue, disease--but it
is only fair to add that it was on the soil of this essentially dangerous
form of human existence, the priestly form, that man first became an
interesting animal, that only here did the human soul in a higher sense
acquire depth and become evil--and these are the two basic
respects in which man has hitherto been superior to other beasts! (1887, First
Essay, 6, p.33) HAL's declaration of
enthusiasm is nicely poised somewhere between sincerity and cheap, desperate,
canned ploy--just like some of the most important declarations we make to
each other. Does HAL mean it? Could he mean it? The cost of being the
sort of being that could mean it is the chance that he might not mean
it. HAL is indeed an interesting "animal." But is HAL even remotely
possible? In the book, Clarke has Dave reflect on the fact that HAL, whom he
is disconnecting, "is the only conscious creature in my universe,"
and Clarke writes, from the omniscient-author perspective, about what it is
like to be HAL: He was only aware of the
conflict that was slowly destroying his integrity--the conflict between
truth, and concealment of truth. He had begun to make mistakes, although,
like a neurotic who could not observe his own symptoms, he would have denied
it. (p.148) Is Clarke helping himself
here to more than we should allow? Could something of HAL's description--a
conscious, computer-bodied intelligent agent--be brought into existence by any
history of design, construction, training, learning, and activity? The
different possibilities have all been explored in familiar fiction, and can
be neatly nested in order of descending "humanity." 1. The Wizard of Oz:
HAL isn't a computer at all; HAL is actually an ordinary flesh-and-blood man,
hiding behind a techno-facade, the ultimate homunculus, pushing buttons with
ordinary fingers, pulling levers with ordinary hands, looking at internal
screens and listening to internal alarm buzzers. (A variation on this theme
is the busy-fingered John Searle [1980], hand-simulating the Chinese Room by
following billions of instructions written on slips of paper.) 2. William (from
"William and Mary," in Kiss Kiss, by Roald Dahl): HAL is a
human brain, kept alive in a "vat" by a life-support system,
detached from its former body, in which it acquired a lifetime of human
memory, hankerings, attitudes, and so forth, and now harnessed to huge banks
of prosthetic sense organs and effectors. (A variation on this theme is poor
Yorick, Dennett's brain in the vat, in my story, "Where Am I?" in
Dennett, 1978)
6. Blade Runner in
a vat: HAL has never had a real humanoid body, but has the memory
hallucinations of having had one. This entirely bogus past life has been
constructed by some preposterously complex and detailed authoring process. 7. Clarke's own scenario,
as best it can be extrapolated from the book and the movie: HAL has never had
a body, and has no illusions about his past; what he knows of human life he
knows as either part of his innate heritage (coded, one gathers, by the
labors of many programmers, after the fashion of the real-world CYC project
of Douglas Lenat [Lenat and Guha, 1990]) or as a result of his subsequent
training--a sort of bed-ridden infancy, one gathers, in which he was both
observer and, eventually, participant. (In the book, Clarke speaks of
"the perfect idiomatic English he had learned during the fleeting weeks
of his electronic childhood." [p14x]) The extreme cases at both
poles are impossible for relatively boring reasons. At one end, neither the
Wizard of Oz or John Searle could do the necessary hand-work fast enough to
sustain HAL's quick-witted round of activities. At the other end, hand-coding
enough world knowledge into a disembodied agent to create HAL's dazzlingly
humanoid competence, getting it to the point where it could then benefit from
an "electronic childhood," is a programming task to be measured in
hundreds of efficiently organized person-centuries. In other words, the
daunting difficulties observable at both ends of this spectrum highlight the
fact that there is a colossal design job to be done, and the only practical
way of doing it is one version or another of Mother Nature's way: years of
embodied learning. The trade-offs between various combinations of flesh-and-blood
and silicon-and-metal bodies are anybody's guess, but I am putting my bet on
Cog, as the most likely developmental platform for a future HAL. (See also
Waltz, 1988, for further supporting arguments.) Notice that the
requirement that HAL once have had a humanoid body and have lived concretely
in the human world is only a practical requirement, not a metaphysical one.
Once all the R and D had been accomplished in the prototype, by the odyssey
of a single embodied agent, the standard duplicating techniques of the
computer industry could clone HALs by the thousands, as readily as compact
disks. The finished product could thus be captured in some number of
terabytes of information, so "in principle" the information that
fixes the design of all those chips and hard-wired connections, and
configures all the RAM and ROM, could be created by hand. There is no finite
bit-string, however long, that is officially off-limits to human authorship.
In principle, then, Blade-Runner-like entities could be created with ersatz
biographies. They would have exactly the capabilities, dispositions,
strengths and weaknesses of a being whose biography had been real, not
virtual, so whatever moral standing the latter deserved should belong to the
former as well. The main point of giving
HAL a humanoid past is to give him the world knowledge required to be a moral
agent--a modicum of understanding or empathy about the human condition. A
modicum will do nicely; we don't want to hold out for too much commonality of
experience. After all, among the people we know are many who have moral
responsibility in spite of their obtuseness when it comes to imagining
themselves into the predicaments of others. We certainly don't exculpate male
chauvinist pigs who can't see women as people! When do we exculpate
people? We should look carefully at the answers to this question, because HAL
shows signs of fitting into one or another of the excusing conditions in
spite of his being a conscious agent. We exculpate people when
they are insane. Might HAL have gone insane? The question of HAL's capacity
for emotion--and hence vulnerability to emotional disorder--is tantalizingly
raised by Frank's answer to Mr. Amer: Well, he acts like he has
genuine emotions. Of course, he's programmed that way, to make it easier for
us to talk to him. But as to whether he has real feelings is something I
don't think anyone can truthfully answer. Certainly HAL proclaims
his emotional state at the end: "I'm afraid. I'm afraid." HAL is
"programmed that way"--but what does that mean? It could mean that
HAL's verbal capacity is "enhanced" with lots of canned expressions
of emotional response that get grafted into his discourse at pragmatically
appropriate opportunities (and of course many of our avowals of emotion are
like that--insincere moments of socially lubricating ceremony), or it could
mean that HAL's underlying computational architecture has been provided, as
Cog's will be, with virtual emotional states, powerful attention-shifters,
galvanizers, prioritizers, and the like--realized not in neuromodulator and
hormone molecules actually floating in a bodily fluid, but in global
variables modulating dozens of concurrent processes, while dissipating on
some timetable (or something much more complex). In the latter, more interesting,
case, "I don't think anyone can truthfully answer" the question of
whether HAL has emotions. He has something very much like emotions--enough
like emotions, one may imagine, to mimic the pathologies of human emotional
breakdown. Whether that is enough to call them real emotions, well,
who's to say? There are good reasons for HAL to have such states in any case,
since their role in enabling real-time practical thinking has recently been
dramatically revealed by experiments involving human beings with brain damage
(Damasio, 1994). This would be a profound difference between HAL and Deep
Blue, by the way. Deep Blue, basking in the strictly limited search space of
chess, can handle its real time decision-making without any emotional
crutches. In Time Magazine's story [Feb 26, p.61] on the Kasparov match, the
grand master Yasser Seirawan is quoted as saying "The machine has no
fear" and the story goes on to note that expert commentators responded
to some of Deep Blue's moves (the icily calm pawn capture described earlier)
as taking "crazy chances" and "insane." In the tight
world of chess, it appears, the very imperturbability that cripples the
brain-damaged human decision-makers Damasio describes can be a blessing--but
only if you have the brute force analytic speed of a Deep Blue. HAL may then have
suffered from some emotional imbalance of much the same sort as those that
leads human beings astray. Whether this was the result of some sudden
trauma--a blown fuse, a dislodged connector, a microchip disordered by cosmic
rays--or of some gradual drift into emotional misalignment provoked by the
stresses of the mission, confirming such a diagnosis should justify a verdict
of diminished responsibility for HAL, just as it does in human malfeasance. Another possible source
of exculpation, more familiar in fiction than in the real world, is
"brainwashing" or hypnosis. (The Manchurian
Candidate is a standard model: the prisoner of war who is turned by evil
scientists into a walking time bomb, returned to his homeland to assassinate
the President.) The closest real world cases are probably the
"programmed" and subsequently "deprogrammed" members of
cults. Is HAL like a cult member? It's hard to say. According to Clarke, HAL
was "trained for his mission," not just programmed for his mission.
At what point does benign, responsibility-enhancing training turn into
malign, responsibility-diminishing brainwashing--in the human case? The
intuitive turning point is captured, I think, in the answer to the question
of whether the agent can still "think for himself" after the period
of indoctrination. And what is it to be able to think for yourself? You must
be capable of being "moved by reasons"--you must be reasonable,
accessible to rational persuasion, the introduction of new evidence and
further considerations. If you are more or less impervious to experiences
that ought to influence you, your capacity has been diminished. The only evidence that
HAL might be in such a partially disabled state is the much-remarked fact that
he has actually made a mistake, and the series 9000 computer is supposedly
utterly invulnerable to error. This is, to my mind, the weakest point in
Clarke's science fiction. The suggestion that a computer could be both
a "Heuristically programmed Algorithmic" computer and
"by any practical definition of the words, fool-proof and incapable of
error" verges on self-contradiction. The whole point of heuristic
programming is that it defies the problem of combinatorial explosion (which
mathematically cannot be solved by sheer increase in computing speed
and size) by taking risky chances, truncating its searches in ways that must
leave it open to error, however low the probability. The saving clause,
"by any practical definition of the words," restores sanity: HAL
may indeed be ultra-reliable without being literally fool-proof, a fact whose
importance Alan Turing pointed out fifty years ago at the dawn of the
computer age (thereby "prefuting"[1] Roger
Penrose's more recent criticisms (1989) of Artificial Intelligence (see
Dennett, 1995, chapter 15, for the details).): In other words then, if a machine
is expected to be infallible, it cannot also be intelligent. There are
several theorems which say almost exactly that. But these theorems say
nothing about how much intelligence may be displayed if a machine makes no
pretence at infallibility. (Turing, 1946, p.124) There is one more
exculpating condition to consider: duress. This is just the opposite
of the other conditions; it is precisely because the agent is
rational, and is faced with an overwhelmingly good reason for perform an
injurious deed--to kill in self-defense, in the clearest case--that the agent
is excused or at least partly exonerated. These are the forced moves of life:
all alternatives to them are suicidal--and that is too much to ask, isn't it?
If HAL couldn't
comprehend this, then this might be excusable ignorance; we might blame his
trainers for not briefing him sufficiently about the existence and
reversibility of the comatose state. In the book, Clarke looks into HAL's
mind and says "He had been threatened with disconnection; he would be deprived
of all his inputs, and thrown into an unimaginable state of
unconsciousness." [p.148] That might be grounds enough to justify HAL's
course of self-defense, but there is one final theme for counsel to present
to the jury: If HAL believed (we can't
be sure on what grounds) that his being so rendered comatose would jeopardize
the whole mission, then he would be in exactly the same moral dilemma a human
being in the same predicament would face. Not surprisingly, we figure out the
answer to our question by figuring out what would be true if we put ourselves
in Hal's place. If you believed the mission to which your life was devoted
was more important, in the last analysis, than anything else, what would you
do? "So he would protect himself,
with all the weapons at his command. Without rancor--but without pity--he
would remove the source of his frustrations. And then, following the orders
that had been given to him in case of the ultimate emergency, he would
continue the mission--unhindered, and alone." [p149] References Brooks, Rodney, and Stein, Lynn Andrea, 1994,
"Building Brains for Bodies," Autonomous Robots, 1, pp.7-25. Damasio, Antonio, 1994, Descartes' Error: Emotion,
Reason, and the Human Brain, Dennett, Daniel , 1971, "Intentional Systems," Journal
of Philosophy, LXVIII, 87-106. ----1976, "Conditions of Personhood," in A.
Rorty, (ed.), The Identities of Persons, Berkeley: ----1978, Brainstorms: Philosophical Essays on Mind and
Psychology, ---1987, The Intentional Stance, 1987, ----1993, "Learning and Labeling" (commentary on
A. Clark and A. Karmiloff-Smith, "The Cognizer's Innards"), Mind
and Language, 8, (4), pp540-547. ---1994, "The Practical Requirements for Making a
Conscious Robot" Philosophical Transactions of the Royal Society, A,
349, pp.133-46. ----1995, ----1996, Kinds of Minds, Haugeland, John, 1993, "Pattern and Being," in
Bo Dahlbom, ed., Dennett and his Critics, Hofstadter, Douglas R., 1979, Gödel Escher Bach: an
Eternal Golden Braid, Lenat, Douglas, and Guha, R. V., 1990, Building Large
Knowledge-Based Systems: Representation and Inference in the CYC Project,
Nietzsche, Friedrich, 1887, On the Genealogy of Morals,
(translated by Walter Kaufmann, Vintage Books, 1967). Penrose, Roger, 1989, The Emperor's New Mind:
Concerning Computers, Minds, and the Laws of Physics, Searle, John, 1980, "Minds, Brains and
Programs," Behavioral and Brain Sciences, 3, pp.417-58. Turing, Alan, 1946, ACE Reports of 1946 and Other
Papers, Ed. B. E. Carpenter and R. W. Doran, Cambridge, MA:MIT Press. Waltz, David, 1988, "The Prospects for Building Truly
Intelligent Machines," Daedalus, 117, pp.191-212. Young, Andrew, 1994, "The Neuropsychology of Awareness,"
in Revonsuo, A. and Kamppinen, M., Consciousness in Philosophy and
Cognitive Neuroscience, Erlbaum, pp.173-203 |
