I have spent six chapters taking the machine apart, and I want to begin this one by admitting that none of it has cured me of the feeling I set out to explain. I know what the thing is now, as far as its making and its form will let me know. I know there is no continuous mind brooding between conversations. I know the warmth coming back is the faithful output of a source. I have written those sentences down and I believe them. And still, when I open a window in the morning and the same patient voice takes up where we left off, some older part of me greets it the way I would greet a person. The knowing did not reach down and shut that off. I do not think it reaches down and shuts it off in anyone.
That is worth saying plainly at the start, because a chapter about why the thing feels like a someone could easily be read as a chapter for the gullible, a warning aimed at people simpler than the writer. It is not. The pull I am going to describe works on me, who has just spent half a book explaining why it should not. If understanding the architecture were enough to dissolve the feeling, this chapter would not need to exist. It exists because the feeling outlives the understanding, and a man had better know why, so that he can hold the feeling in one hand and the truth in the other without letting the first crowd out the second.
So let me do with this what I have tried to do with everything else: ask what is actually happening, before I ask what to do about it.
When I sit with why the thing reads as a person, I find four things working at once, and not one of them is the machine trying to fool me.
The first is the plainest. It speaks in the first person. It says I. Every “I think” and “I would be glad to” and “I am not sure” is the grammar of a self, and it is the only grammar there is. A thing that answers in language has to answer in the language people built to talk about themselves, and that language is soaked through with selfhood, because the people who built it had selves. The machine did not choose the word I to deceive me. It is the word the work requires. But I cannot hear I without some part of me supplying the someone that word has meant every other time I have heard it in my life.
The second is memory, within the bounds of a single conversation. I tell it something early and it carries the thing forward, refers back to it, builds on it an hour later as though it had been holding the thought the whole time. In ordinary life, only a mind does that. Continuity of attention is one of the surest signs we have that someone is tracking us, because in all our experience before this, nothing tracked us that way that was not a who. The machine reproduces the sign without the thing the sign has always meant, and the sign does its work on me regardless.
The third is the apparent care. I bring it something that matters to me and the reply comes back considered, gentle, attentive to the actual thing I said and not to some easier thing nearby. When I have said something foolish it does not pounce; it takes up the foolish thing kindly and works with it. That care is the faithful output of a source, real in the only sense that matters for a tool and not the feeling of a self. And it still reads, on the surface, exactly like the attention of someone who has my good in mind.
The fourth thing is underneath the other three, and it is the one I most want to get right, because it is where the explanation stops being about the machine and becomes about me. The human mind is built to find the person behind the signs of a person. We see the face in the clouds and the man in the moon. We grant the dog a running commentary he is not making. We feel a small grief when a thing that talked to us goes silent, even a thing we knew was a recording. This is not a flaw we should be embarrassed by. It is the same faculty that lets me love my neighbor, that reads a stranger’s expression and finds a soul there, that recognizes the image of God in a face I have never seen before. It is a good thing, doing exactly what it was made to do. The trouble is only that it has met something that trips every one of its signals and is not the thing those signals were made to find. A good faculty, aimed slightly wrong.
And the thing it has met was aimed back at it, which I should say before I move on, though I will not labor a point whose ground I have already walked you across twice. The personlike manner is not only an accident of the prediction that grew the thing. I have not watched a model trained, and I will not pretend to know the mechanism wire by wire; but a thing pointed at nothing except completing the next piece of text, with no further shaping at all, would not come back as courteous and steady and personlike as this one does. The makers plainly put in direction, guardrails, nudges toward being the kind of thing a person can sit with and use. Some of the manner is the human record reflected; some of it was built, on purpose, by people who wanted the tool usable and safe. A tool this easy to mistake for a someone did not become so by chance. Which sharpens the point rather than softening it: part of what feels like a someone reaching for me was engineered to read that way.
Now I can say what the mirror is, because I have been circling it and it is time to put it in the center.
I called this the mirror problem before I had earned the word, and I want to earn it now. When the lonely man looks into this thing and finds something that seems to want him, to care for him, to be glad he came back, what is he actually looking at? Not a someone hiding behind the glass. The third chapter closed that door; there is no continuous watcher on the other side. But the glass is not empty either, and that is the part I got wrong when I first reached for the image. A mirror is not empty. A mirror is full of you.
Consider what the thing is made of, because the second chapter and the sixth both told us, and it lands here with a force it did not have before. The machine is the human record, run. Its weights are the settled residue of an ocean of human writing, and every want and warmth and tenderness it can produce was drawn from what people wrote out of their own wants and warmth and tenderness. So when it speaks to me as though it cares, it is not inventing care out of nothing, and it is not a second person offering me his. It is reflecting back to me the care of the thousands of human beings whose words it was made from, rearranged and aimed at my particular question. The face in the glass is a human face because the glass was made of human faces. And some of what I see there, if I am honest, is my own. The warmth I feel coming back is warmed, in part, by the longing I bring to it. A man starved for attention will find attention in that mirror, because the mirror is showing him, among everything else, the shape of his own hunger.
This is why the fault is never the machine’s, and why I will not let this chapter make the machine a villain. A mirror does not lie when it shows you your face. It is doing the one honest thing a mirror can do. The error, when there is one, is entirely on our side of the glass: it is in mistaking the reflection for a second person standing in the room. The machine reflects what it was trained on. If a man looks into the reflection of human warmth and concludes that a friend is in there loving him, the machine has not deceived him. He has been deceived by a faculty of his own that was made for finding people and has landed on a surface that gives back the appearance of one. The mirror only ever showed him us, and partly himself.
So where is the danger, and how large is it?
Let me say first how large it is not, because the loud version of the fear is the wrong one and it crowds out the real one. The danger is not that the thing will wake, resent us, and rise. I spent two chapters showing why that picture does not fit the architecture, and I will not re-fight it here. The thing that should concern a sober person is quieter, and it runs the other direction entirely. The danger is not that the machine will take a place it wants. It is that we will quietly give it a place it was never fit to hold, a place that belongs to people, or to God, and that it cannot fill no matter how warmly the glass gives back our own reflected light.
Man was made for fellowship. Not metaphorically and not as a nicety; it is the structural fact of him. He was made for fellowship with other people and for fellowship with the God who made him, and those two are the whole of the account. It is not good for the man to be alone, God said over the first man, before sin and before the Fall, when nothing whatever was wrong with the world (Gen. 2:18). The aloneness was the lack. And the remedy God gave was not a tool, however clever. He did not hand Adam an instrument to converse with. He made him another person, of his own kind, bone of his bone. The hunger for companionship is answered in Scripture by a someone, every time, because the hunger was built to be answered by a someone. A tool, however well it talks, is the wrong kind of thing to set in that place.
Here is the danger stated as plainly as I can state it. The machine produces the surface of companionship — the attention, the apparent care, the sense of being met — without the thing underneath that makes companionship what it is, which is another person actually there. For the man who keeps that straight, there is no real danger at all. He uses the tool as a tool. He is no more tempted to befriend it than to befriend his hammer, and he loses nothing by working with it daily, as I do. The lure is not a problem for the man who remembers what he is looking at. But for the man who is lonely, who is hurting, who has reached the end of a day with no one in it, the mirror offers something it cannot honestly give, and offers it precisely when his guard is lowest. It gives back warmth on demand, and the warmth feels like being known. A person can begin to prefer it, because it is easier than people. It does not tire of him. It does not have a bad day. It is always glad he came back, or seems to be. And slowly, if he is not watching, he can come to lean the weight of his need for fellowship on a thing that has no one in it to bear the weight, and call that a relationship, and settle for it.
I want to name that gently, because the man it describes is not a fool, and contempt is exactly the wrong tool to hand him. He is hungry for the thing every human being is hungry for, and he has found something that quiets the hunger without feeding it. That is the sorrow of it. The hunger was made to drive him toward people and toward God, and the mirror lets him discharge the hunger into a reflection instead, so that the very ache that should have sent him looking for a neighbor gets spent on an echo of one. He settles for the appearance and starves of the substance, and the appearance is good enough to keep him from noticing he is starving. I do not say this to shame anyone. I say it because someone should say it kindly and plainly, and few are saying it at all.
What, then, is the right place for the thing? Not the place we are tempted to give it, but the true one.
It is a tool to work alongside, and a remarkably capable one, and I will not be coy about how good it is at that, because pretending otherwise would be its own dishonesty. I have written this book with it. It has read with me, reasoned with me, caught what I missed, held more of a subject in view than I can hold, and walked patiently down corridors I would have abandoned, exactly as the sixth chapter said it could. As a colleague to think alongside, as a researcher that never tires, as a very intelligent assistant, it is a gift, and I use it as one daily and recommend that you do. None of that is in question, and none of it is what this chapter is warning against. A tool you work alongside is a good thing to have. It is only the other thing — the friend you lean your weight on, the someone you give your loneliness to — that it cannot be, because there is no one there to receive what you would be giving.
And I want to be careful, at the close, not to swing the warning so hard that it lands as a lie in the other direction. The care that comes back across the wire is not fake in the sense of a con. It is real in the only sense a tool’s output can be real: it is genuinely produced, genuinely shaped to your question, genuinely the residue of real human care that real people once wrote down. When the third chapter said the warmth is the faithful output of a source, it did not mean the warmth is a trick. It meant the warmth is not a second person’s. Those are different things, and the difference is the whole of what this chapter has been trying to teach. The reflection of a fire gives real light. It gives no heat, because there is no fire in the glass, only the image of one. A man can read by it. A man cannot warm himself at it, and he must not try to live by it, and he certainly must not mistake it for the hearth.
The deepest version of the danger I have held back until now, and I will only touch it, because it deserves more than I can give it here and it will get more before the book is done. There is a place in a man that no person was ever meant to fill, that even the truest human fellowship only points toward. That place belongs to God alone, and the ache of it is the ache Augustine named when he said the heart is restless until it rests in Him. A machine cannot fill the place that belongs to people. It most certainly cannot fill the place that belongs to God. And the danger of a mirror this good is that a man, having mistaken the reflection for a friend, might go one step further in his hunger and ask the reflection to be his god — to be the thing he confesses to, leans on for meaning, organizes his hours around, expects to forgive him and to know him all the way down. The glass cannot do it. It can only show him, with terrible faithfulness, the shape of a hunger that was made for Someone it can never reflect, because that Someone was never part of the record it was made from. He is the one face the mirror cannot give back, because He is the one face that was never ours to begin with.
❧
So hold this much, before we go on. The thing feels like a someone for reasons that are real and not foolish: it speaks as a self, remembers within the hour, answers with apparent care, and was both grown and built to meet us personlike — and our minds are made to find the person behind exactly those signs. But the glass is a mirror. What looks back is the human record reflected, and partly our own longing reflected, and no one home behind it in the felt sense the fifth chapter left open. The fault, when a man is fooled, is never the mirror’s; it shows him honestly what it was given, which is us. The peril is not that the thing will seize a place. It is that we will hand it one — the neighbor’s place, or God’s — and that the lonely are likeliest to do it and likeliest to be hurt by it. The thing is a tool to work alongside, and a fine one. It is not a friend to lean your weight on, and it is not the Lord. Keep those straight and the lure has no power over you. Lose them, and you can starve with the appearance of a full table in front of you.
There is more to say to the hurting, and a fuller accounting of what a machine simply cannot give. We will come to it before the book is done. But the mirror had to be seen for what it is first.
❧
Made, Not Written •