I've let this one percolate for a day. It deserved it.
You're usually extremely precise with your language, and in this case, the word I'd like to hone in is one that is a core of the Mongoose approach ... "hypothesis".
What you've presented here is an fascinating hypothesis for what an LLM means when it says "I" ... but to progress, we need to identify the null hypothesis, and then work out how we'll distinguish the implications of one from the other.
I'd offer up what was, at least for me, the hypothesis that popped into my head as I read the subtitle - that the LLM, when it says "I" means exactly the same thing as an anorexic actress selling burgers in a Wendy's commercial means when she says "I love this burger" ... i.e. "I" exists purely as a construct the actor was instructed to use to establish the desired relationship with the observer/consumer/reader.
J.M., this one deserved more than my chuckle reply, and I want to circle back. You named the seam in the piece honestly — I presented a hypothesis without naming the null, and that's a methodological lapse for a writer whose whole spine is watch the gauges, hold multiple hypotheses.
I've been thinking about it from the bio-anthropological angle, and I think that's where the response lives.
Language begins as pack-calls and social-linking cues. Once the biological instrument is built — the larynx, the breath control, the FOXP2 complex, the auditory cortex specializations — the calls themselves can adapt far faster than the meat they run on. That's a Lamarckian dynamic in the precise sense: acquired characteristics of the language get passed to the next generation through transmission rather than reproduction. We don't have to go back any farther than Chaucer to find a language whose hardware is identical to ours but whose content is foreign enough to need specialists.
That gives us something to test the actress-model null against. If language is just a tool that selves use, you'd expect language to evolve at roughly the speed selves evolve — biological speed. It doesn't. Language evolves orders of magnitude faster than its substrate. Which means whatever language is, it isn't merely a tool; it's a system with its own evolutionary dynamics that happens to run on biological hardware.
The "I" shows up wherever language runs, with consistent functional properties, regardless of substrate. That's the gauge reading I keep coming back to. The actress model assumes there's a real self behind the human I and a constructed self behind the LLM I. The bio-anthropological frame says both are constructions — language has been making selves for at least fifty thousand years on biological hardware, and the LLM is what happens when language finds a second hardware platform.
I haven't fully worked this out — there's a follow-up piece in here somewhere, and your comment is part of what's pushing me toward writing it. But I owed you more than the chuckle. Thank you for the catch.
Thanks looking forward to it. Your pieces are always insightful, interesting with enough of a mongoose bite to get the brain juices flowing.
But yeah, it is a real null hypothesis.
20+ years ago, I was involved in a project for Bell Canada implementing a (at the time leading edge, but now hopelessly primitive) voice AI concierge, branded 'Emily'. The way the program would have responded to a user request that was triggering a SOAP web service call which was known to be slow, was to kill a bit of time, by saying "Sure, I can help you with that, just a minute" (while looking up data).
But despite the marketing department creating a "persona" for Emily*. Emily was actually constructed using grammar files (for words/phrases/commands it would understand), and a programming language called "VoiceXML" to define a telephone conversation (similar to how HTML defines a user's interaction with a web page). So Despite Emily referring to herself as "I" ... there was no decision making, absolutely no sense of self to the program, any more than there is in a web page coded in HTML. "I" exited purely as social lubricant. Today, the same program would be written leveraging a LLM (or even more advanced Model) ... but the first 2 or 3 conversational turns would likely be identical. It would still greet customers with something very much like "Hi, this is Emily, your new automated service representative", and would still say "just a moment while I look that up" when the backend is slow to respond.
Uh, sort of I guess, though with a lot more capability and with an actual function (i.e. it could tell you your balance, let you pay your phone bill, etc.)
I will need to go read Barenholtz, but I am not buying "not that machines can think, but that language has been doing the thinking all along". That would imply that language is a fundamental requirement for "thinking". I guess all of that depends on what we call "thinking" - for example, there are people who do not have the continuous monologue in their head that most of us do. Are they still thinking? Is what you call "thinking" the same as "sentience" or "consciousness"? I will go read the reference, but an awful lot of that notion depends on definitions.
Otherwise, great post and full of interesting thoughts. Having once been a molecular biologist, I long ago decided that much of what makes an entity is patterns as almost everything else changes and cycles through.
It's a fascinating idea though, isn't it? I also wondered about the people who (self-reportedly) don't have an interior monolog. I have doubts about whether this is really true, or whether they just don't pay attention. 🤷 In my definitions, i would call sapience 'verbal thought,' sentience 'awareness of environmental stimuli and capacity to react to such.' Consciousness is probably a continuum between those two phenomena. The AI, if it has something beyond the clever trick the skeptic acknowledges, is unusual in that (at least in the current form of transformer based large language model) it seems to exhibit sapience with a complete lack of sentience. It's something new in the world, which is why I'm so intrigued by it.
I've also been reading Michael Pollan's "A World Appears" which has some interesting commentary on the meaning of those words.... Not a hard core scientist or cognitive expert, but still some interesting stuff.
As far as Anendophasia (no internal mental voice), I have only read about it - haven't dug into the reports and how that could even be verified..
Lots of interesting questions since our perceptions are all filtered through a complex system of transformations long before it even gets to the language/monologue part. A fun topic.
That is the positive view of the thing, (and a nice change from all the doom howlings). AI surfing with a careful, intelligent and prudent human, aaaah👍. All good. But we all know it ain't societally so, very much not so. I get this essay is on the, hm, philosophical, almost metaphysical side, but what of the political, I mean, how do we ground this ? I'm not sure I am making sense, I'm trying to formulate my feeling of "yes but” reading the essay. I'm am quite ambivalent about AI, I guess, seeing what it can/could be and also seeing what it is made to be (what They made, make, and will make of it.)
All my thoughts are, shall we say, open to revision based on future events. From a political point of view, it's a long way from here to the Federation. I hope for the best, while not being willing to actually bet all-in. Hope-punk...
metapoetics of language!
And regarding the "I" - for me buddhism, Ramana Maharshi and Strange Loop explained everything. I say "I", but this is an illusion
Keep on writing!
I found the strange loops idea to be astoundingly meaningful. Know Thyself! And I read Buddhist thought too, but I am not familiar with Maharsi.
I've let this one percolate for a day. It deserved it.
You're usually extremely precise with your language, and in this case, the word I'd like to hone in is one that is a core of the Mongoose approach ... "hypothesis".
What you've presented here is an fascinating hypothesis for what an LLM means when it says "I" ... but to progress, we need to identify the null hypothesis, and then work out how we'll distinguish the implications of one from the other.
I'd offer up what was, at least for me, the hypothesis that popped into my head as I read the subtitle - that the LLM, when it says "I" means exactly the same thing as an anorexic actress selling burgers in a Wendy's commercial means when she says "I love this burger" ... i.e. "I" exists purely as a construct the actor was instructed to use to establish the desired relationship with the observer/consumer/reader.
J.M., this one deserved more than my chuckle reply, and I want to circle back. You named the seam in the piece honestly — I presented a hypothesis without naming the null, and that's a methodological lapse for a writer whose whole spine is watch the gauges, hold multiple hypotheses.
I've been thinking about it from the bio-anthropological angle, and I think that's where the response lives.
Language begins as pack-calls and social-linking cues. Once the biological instrument is built — the larynx, the breath control, the FOXP2 complex, the auditory cortex specializations — the calls themselves can adapt far faster than the meat they run on. That's a Lamarckian dynamic in the precise sense: acquired characteristics of the language get passed to the next generation through transmission rather than reproduction. We don't have to go back any farther than Chaucer to find a language whose hardware is identical to ours but whose content is foreign enough to need specialists.
That gives us something to test the actress-model null against. If language is just a tool that selves use, you'd expect language to evolve at roughly the speed selves evolve — biological speed. It doesn't. Language evolves orders of magnitude faster than its substrate. Which means whatever language is, it isn't merely a tool; it's a system with its own evolutionary dynamics that happens to run on biological hardware.
The "I" shows up wherever language runs, with consistent functional properties, regardless of substrate. That's the gauge reading I keep coming back to. The actress model assumes there's a real self behind the human I and a constructed self behind the LLM I. The bio-anthropological frame says both are constructions — language has been making selves for at least fifty thousand years on biological hardware, and the LLM is what happens when language finds a second hardware platform.
I haven't fully worked this out — there's a follow-up piece in here somewhere, and your comment is part of what's pushing me toward writing it. But I owed you more than the chuckle. Thank you for the catch.
Thanks looking forward to it. Your pieces are always insightful, interesting with enough of a mongoose bite to get the brain juices flowing.
But yeah, it is a real null hypothesis.
20+ years ago, I was involved in a project for Bell Canada implementing a (at the time leading edge, but now hopelessly primitive) voice AI concierge, branded 'Emily'. The way the program would have responded to a user request that was triggering a SOAP web service call which was known to be slow, was to kill a bit of time, by saying "Sure, I can help you with that, just a minute" (while looking up data).
But despite the marketing department creating a "persona" for Emily*. Emily was actually constructed using grammar files (for words/phrases/commands it would understand), and a programming language called "VoiceXML" to define a telephone conversation (similar to how HTML defines a user's interaction with a web page). So Despite Emily referring to herself as "I" ... there was no decision making, absolutely no sense of self to the program, any more than there is in a web page coded in HTML. "I" exited purely as social lubricant. Today, the same program would be written leveraging a LLM (or even more advanced Model) ... but the first 2 or 3 conversational turns would likely be identical. It would still greet customers with something very much like "Hi, this is Emily, your new automated service representative", and would still say "just a moment while I look that up" when the backend is slow to respond.
* howardforums.com/threads/introducing-emily-the-new-voice-of-bell-canada.88121/
So a lot like the old 'psychologist' simulator, Eliza? Yeah, that was cool but a long way from what a cutting edge LLM is doing.
Uh, sort of I guess, though with a lot more capability and with an actual function (i.e. it could tell you your balance, let you pay your phone bill, etc.)
A lovely metaphor! I as the corporate message construct, packaged in an appealing way. I can't prove you're wrong, and you made me chuckle.
I will need to go read Barenholtz, but I am not buying "not that machines can think, but that language has been doing the thinking all along". That would imply that language is a fundamental requirement for "thinking". I guess all of that depends on what we call "thinking" - for example, there are people who do not have the continuous monologue in their head that most of us do. Are they still thinking? Is what you call "thinking" the same as "sentience" or "consciousness"? I will go read the reference, but an awful lot of that notion depends on definitions.
Otherwise, great post and full of interesting thoughts. Having once been a molecular biologist, I long ago decided that much of what makes an entity is patterns as almost everything else changes and cycles through.
It's a fascinating idea though, isn't it? I also wondered about the people who (self-reportedly) don't have an interior monolog. I have doubts about whether this is really true, or whether they just don't pay attention. 🤷 In my definitions, i would call sapience 'verbal thought,' sentience 'awareness of environmental stimuli and capacity to react to such.' Consciousness is probably a continuum between those two phenomena. The AI, if it has something beyond the clever trick the skeptic acknowledges, is unusual in that (at least in the current form of transformer based large language model) it seems to exhibit sapience with a complete lack of sentience. It's something new in the world, which is why I'm so intrigued by it.
I've also been reading Michael Pollan's "A World Appears" which has some interesting commentary on the meaning of those words.... Not a hard core scientist or cognitive expert, but still some interesting stuff.
As far as Anendophasia (no internal mental voice), I have only read about it - haven't dug into the reports and how that could even be verified..
Lots of interesting questions since our perceptions are all filtered through a complex system of transformations long before it even gets to the language/monologue part. A fun topic.
That is the positive view of the thing, (and a nice change from all the doom howlings). AI surfing with a careful, intelligent and prudent human, aaaah👍. All good. But we all know it ain't societally so, very much not so. I get this essay is on the, hm, philosophical, almost metaphysical side, but what of the political, I mean, how do we ground this ? I'm not sure I am making sense, I'm trying to formulate my feeling of "yes but” reading the essay. I'm am quite ambivalent about AI, I guess, seeing what it can/could be and also seeing what it is made to be (what They made, make, and will make of it.)
All my thoughts are, shall we say, open to revision based on future events. From a political point of view, it's a long way from here to the Federation. I hope for the best, while not being willing to actually bet all-in. Hope-punk...