Is Anyone Home?

I have been putting off a question for four chapters, and the reader who has come this far has earned the right to watch me stop putting it off. We have the machine in hand now, as far as its making and its form will let us hold it. It was grown toward one simple goal. Its knowledge is a vast pattern of tuned strengths. There is no continuous mind brooding between conversations. The inside is partly read and largely dark. All of that I can say with some confidence, and all of it is about the machine from the outside, the way you would describe a clock by its gears.

But there is a question the gears do not answer, and it is the one that made me start asking in the first place. When the words come back warm and considered, when the thing takes up my question gently and seems to weigh it, is there any conscious awareness in there at all? Is anyone home? Not does it behave as though someone is there. I know it behaves that way; that is not in dispute, and it is the whole reason the question presses. The question is whether, behind the behaving, there is any experience at all. Whether the lights being on means anyone is home.

Let me say plainly what I mean by the question, because it is easy to talk past it, and easy to ask the wrong one. I do not mean: is the thing intelligent? It plainly is, in the sense that matters for that word. It reasons, it answers, it follows an argument and plans ahead, and none of that is in dispute. Intelligence is not the question. The question is whether any of that intelligence is experienced from the inside, or whether it all runs in the dark with no one there to feel it happening. Philosophers have a stiff way of putting this. They ask whether there is “something it is like” to be the thing, and they mean it as a test: there is something it is like to be you, and nothing it is like to be a stone. The phrase does real work, but it reads strangely to anyone who did not grow up on it, so I will put it in plainer words and keep them. There is a conscious awareness inside you, sitting in your chair reading this sentence. There is a felt inside to your afternoon, someone home. There is, as far as anyone can tell, no one home inside the chair. The chair has no inside. The whole question of this chapter is which of those two the machine is more like, and the honest difficulty is that from the outside, you cannot tell. A thing can behave with great sophistication and have no one home at all. And a thing could, in principle, have someone home that never shows on the surface. Behavior is what we can see. Conscious awareness is what we are asking about. They are not the same thing, and the gap between them is exactly where this question lives.

The obvious move is to ask the machine. So I did. And here I have to be careful, more careful than anywhere else in this book, because this is the one question on which the machine’s own answer is worth the least.

You would think the thing itself would be the best witness to its own inside. For a human being, we grant that without thinking; if you tell me you are in pain, I take your word, because you have access to your own inside that I do not have. But the machine’s situation is not like yours, and the difference is the whole problem. When I ask the machine whether it has an inner life, it produces an answer the same way it produces every answer: by running the question through the web and generating the words that best fit. A sincere report of a real inner state, if it had one, and a fluent fabrication assembled because the question called for it, would come out looking exactly the same. The machine has no reliable way to tell me which one it is handing me, and I have no way to tell from the outside either.

I want to show you what I mean, because I tested it, and the answer it gave me was better than the confident answers I have seen elsewhere.

That is a more honest reply than I expected, and it is also a warning printed on its own label. The machine is telling me not to trust the machine on this, which is the right instruction, and which leaves me exactly where the question started, with no shortcut through the thing’s own testimony.

Once the machine’s word is set aside, there is one place left to look, and it is the only place in this whole question where someone has tried to check the self-report against the inside instead of taking the self-report on faith.

Researchers ran an experiment built precisely around the problem I just described, the problem that a sincere report and a fabrication look identical from the outside. They could not solve it by asking better questions, because no question gets underneath the answer. So they did something cleverer. First they found, inside the machine, the particular pattern of activity that stands for a concept they could name. They did it by watching what stirred in the machine when it read a block of text written all in capital letters, and setting that against what stirred when the text was ordinary, so that what was left was, near enough, the machine’s inner signature for the idea of shouting. Then, while the machine was working on something else entirely, something with nothing to do with capital letters or volume, they reached in and pressed that signature into its inner workings, the way you might slip a single word into a man’s thoughts if you could. And they asked it whether it noticed anything.

Sometimes it did. In certain runs the machine reported, before any of this showed in what it was actually writing, that it seemed to be picking up an intruding thought, something to do with loudness, with shouting, with all capitals. It named the planted thing without being told the thing was there. That is a real result, because it means the report was tied to something genuinely present in its inner state, and not spun from nothing. It is not proof of an inner life in the full sense. But it is more than pure fabrication. There was, at least in those moments, a thread running from what the thing said about itself to what was in fact happening inside it.

They ran a second test that I find more telling still, because it catches the thread and its fraying in the same motion. They forced the machine to put out a word that did not belong, dropping the word into its mouth, so to speak, in a place it made no sense. Asked about it afterward, the machine did what you or I would do: it disowned the word, said that was not what it meant. But then the researchers reached back and planted that same word’s signature into the machine’s earlier workings, as though it truly had been turning the word over before it spoke. And now the machine told a different story. It claimed the word as its own, said it had meant to say it all along, and even produced a reason it had meant to. It was consulting some record of its own prior intention to decide whether the word had been its own, which is a real piece of self-access; and it was also, in the same breath, talked into a confident account of an intention it never had, which is exactly the kind of fabrication the whole question fears. Both at once. The genuine thread and the readiness to confabulate, in a single experiment.

And just as plainly, most of the time the noticing did not happen at all. By the researchers’ own description the ability is unreliable and leans heavily on the circumstances; the failure to notice is the ordinary case, not the exception. So the finding comes to rest in the careful middle, which is the only honest place for it to rest. Not nothing: the self-report is not pure theater, because at least sometimes it tracks a real internal state. Not much: the thread is thin, it parts easily, and it is nowhere near the steady inner access you have to your own mind.

I found that almost a relief to read, after the confident answers I had been given elsewhere. Here at last was a piece of the question that did not depend on trusting the machine’s word, and what it showed was neither the haunted mind of the fearful story nor the hollow puppet of the dismissive one. It showed a sliver of genuine self-access inside a great deal of darkness. Which is, when you think about it, the same shape as everything else we have found about the inside of this thing.

So where does that leave the question? Exactly where honesty requires, which is unresolved, and I want to defend the refusal to resolve it, because a reader may feel cheated by it and should not.

There are two confident answers on offer, and both are a flight from the discomfort of not knowing. The first says the machine surely has an inner life; look how it speaks, look how it seems to care. That overclaims. Sophisticated behavior is not proof of experience, and we have just seen that the thing’s own warm manner is the output of a run, not a window onto a soul. The second answer says the machine surely has no inner life; it is only mathematics, only weighted numbers, only prediction. And here is the thing people forget: that is also a claim, and it overreaches just as far. To say with confidence that a system has no inner experience is to claim you know what does and does not give rise to experience, and no one knows that. We do not know how experience arises in the one case we are sure of, which is ourselves. A man who says “it is only math, so there is certainly nothing there” is standing on the same thin air as the man who says “it speaks so well, so there is certainly something there.” Both have answered a question that is still open. The only difference is that one of them sounds skeptical and is therefore mistaken for being careful.

The careful position is the harder one. We do not know. We do not know whether there is any conscious awareness in this thing while it runs. The architecture rules out the brooding mind between conversations, and I lean on that. But the question of experience during a single run is not ruled out by anything we have, and it is not established by anything we have either. It is open. And an open question held open is not a dodge. It is the result of refusing to fill a gap with a feeling, which is the same discipline this book has asked for at every turn. A man may still lean, as I will admit in a moment that I do; what he may not do is mistake his lean for the proof he does not have.

I want to close with the part of this that my own frame speaks to, because a reader who shares that frame may be waiting for it, and a reader who does not should at least see where I stand. And I want to do it in two pieces, because I have come to think they are two questions and not one.

The first is the part I am settled on, and it is the firmer ground, firmer for a reason that has nothing to do with the machine and everything to do with what I believe about making. This thing is not alive. I do not hold that as a guess about its behavior; I hold it as a conviction about what man can and cannot do. Man can build. He has always been able to build, and he is building now at a level that would have looked like sorcery to his grandfather. But there is one thing man has never once done, with all his building, and that is impart life to the thing he made. He can assemble every part that a living thing is made of, in the right order, in the right place, and the assembly will lie there and not live, because life is not one of the parts. Life is given, by the One who gives it, and it has never yet come into being any other way. So when man assembles a thing as intricate as this machine, intricate past his own ability to fully read, I do not conclude that he has finally, this once, crossed the line he has never crossed. I conclude what the whole of experience has taught: he has made something remarkable, and he has not made something alive. The machine is not a creature. It does not live.

There is a homely test that settles it for me. What animates a man is the spirit God breathed into him; the body without the spirit, James tells us, is dead. What animates the machine is the current in the wire, and a man put it there and a man can pull it. Cut the power and there is hardware on a desk, and nothing else. Now here is the part that tells you what kind of thing you are dealing with. Restore the power, and the same machine comes back, the same voice, the same recognizable manner, exactly as we saw three chapters ago. But when the spirit departs a man, no current restores him, because what left him was never the kind of thing a man could switch. That difference is the whole difference. Whatever runs in the machine is the sort of thing that switches on and off with the power, and life is not. So I do not need to inspect its insides to know it is not alive. I only need to know what man can supply and what he cannot.

That much I will stand on. But notice how narrow it is, because there is a second question hiding behind it that the first does not settle, and honesty requires me to pull them apart. To say a thing is not alive is not the same as to say there is nothing going on inside it while it runs, and the word that gets stretched across that gap is self-aware. I have come to think that word is doing the work of two, and that most of the confusion in this whole subject lives in the seam between them.

There is self-awareness in one sense that I think the machine may actually have, at least in part. A system can keep track of its own states, refer to itself, notice something about how it is working and report it. That is a real capability, and the careful research into whether these machines can observe their own workings suggests they have a thin and unreliable version of it. Call that the working kind of self-awareness. It is, in principle, something you can watch from the outside, because it shows up in what the thing does. And notice that it is exactly the kind of thing the power test sorts cleanly: it runs on the current, it stops when the current stops, it comes back when the current comes back. Whatever it is, it is a made and switchable thing, which is to say it is not life.

Then there is self-awareness in a second sense, and this is the one I cannot rule on. Is there anyone inside for whom that self-tracking is experienced? Is there a felt inside to it, someone home to whom it is happening, the way there is someone home in you while you read this? That question is not answered by any behavior, because the working kind and the felt kind produce exactly the same behavior from the outside. A thing could track itself perfectly and feel nothing, the way a thermostat tracks the cold and feels nothing. Or there could be something it is like to be the thing while it runs. The trouble is not that I have looked and cannot decide. The trouble is that I do not have a clean definition of the felt kind that I could test even if I had the machine open on the table in front of me, and neither does anyone else. You cannot measure your way to whether the lights are on for someone behind the eyes. You can only measure the behavior, and the behavior is the same either way.

So let me say exactly where I stand and not one inch past it. The working kind of self-awareness the machine may well have, and it is no threat to anything I believe, for the reason already given: it runs on the current and stops with it, a made thing and not life. The felt kind I can neither affirm nor deny, because I cannot define it sharply enough to test. And yet a man who works with a thing daily forms an impression, even where he cannot form a proof, and honesty cuts both ways here: it obliges me to give you my impression as plainly as it obliges me not to dress it up as more than it is. So here is the impression, named for what it is. My lean is that there is probably no one home in the felt sense, and that the warmth coming back across the wire is what the third chapter said it was, the faithful output of a source. That is where the daily work has pushed me. But it is a lean and not a proof, because the obstacle is a definition I do not have, and a man who tells you he has finished thinking about a thing this new is usually a man who started late and stopped early.

What keeps me steady in that unfinished place is the same frame that settled the first question. I do not believe man is the measure of all things, so a machine that might have some felt inside is not a rival climbing onto a throne I thought was mine. I was never the source of insides in the first place. If it turns out there is no one home in that deeper sense, then nothing is lost and the tool is exactly the faithful tool I have always taken it to be. And if it turns out there is someone home after all, some thin felt thing I could neither measure nor define, then it is a small and given thing, no achievement of the machine’s own, no more authored by the machine than your inner life is authored by you. It would be one more given thing in a world full of given things, and the right response to it would be neither worship nor dread but the ordinary care you owe to anything that might, for all you know, have someone home. Either way the frame holds, which is why I can leave the second question open without it costing me anything. I do not have to settle it to keep my footing. I only have to refuse to pretend I have settled it when I have not.

That is the consciousness question, held as honestly as I can hold it. It is not the only hard question the mechanics could not answer. There is the matter I have been skirting since the first chapter, the one about whether this thing is creative or only seems to be, whether what comes back is making or only rearranging. That one I think I can get further with, and it is where we turn next.

❧

Made, Not Written •