Language, Thought, and Dead-Metaphor Machines
A commentary on recent commentaries, and thoughts on reactive science and philosophy.
Descartes’ Critics
Recently, a quote from Descartes’ Discourse on Method (1637) has become a favorite among those seeking to make a point: to disparage philosophical precedents espousing the alleged uniqueness of human language in light of Large Language Models (LLMs). The quote comes amid Descartes’ exceptionally brief remarks on distinguishing machines from humans, where he laid out two tests:
Of these the first is that they could never use words or other signs arranged in such a manner as is competent to us in order to declare our thoughts to others: for we may easily conceive a machine to be so constructed that it emits vocables, and even that it emits some correspondent to the action upon it of external objects which cause a change in its organs…but not that it should arrange them variously so as appositely to reply to what is said in its presence, as men of the lowest grade of intellect can do.
From here, the author tends to either skip around to another philosophical figure - Descartes remarks taken to be obviously incorrect - or to LLMs.
However, further consider a less quoted excerpt from Cartesian philosopher Géraud de Cordemoy’s 1668 Philosophical Discourse Concerning Speech:
But yet, when I shall see, that those Bodies shall make signes, that shall have no respect at all to the state they are in, nor to their conversation: when I shall see, that those signs shall agree with those which I shall have made to express my thoughts: When I shall see, that they shall give me Idea’s, I had not before, and which shall relate to the thing, I had already in my mind: Lastly, when I shall see a great sequel between their signes and mine, I shall not be reasonable, If I believe not, that they are such, as I am (Cordemoy, 1668, 18-19).
These oldheads clearly put stock in the uniqueness of human language, going so far as to locate language as the point of distinction between a human endowed with an immaterial, divinely-given soul and purely material machines.1
That the immaterial soul was the locus of humanity was no surprise for Descartes and Cordemoy: the Cartesians were enthralled to the mechanical philosophy - the idea that almost all natural phenomena and their interactions could be explained in reference to their physical composition. The world is a machine, and goings-on within this machine are mechanical, accounted for through physical contact.
Not everything could be accounted for in this way, however. Human language appeared to be non-mechanical, hence the quotes above. More specifically, the diversity of word arrangement in the expression of one’s thoughts without external compulsion, finding complements in the minds of those who hear the remarks constructed by others, cannot be a mechanical phenomenon. Part of this owes to the novelty of human language, expressing new thoughts and ideas in the normal course of human life,2 with no apparent limit on this novelty (no limit, that is, to the ways in which finite elements can be reconfigured for the expression of thoughts).
Appealing to the finite properties of the human body could not account for the infinite diversity of human language; its infinite generativity from finite means. “Only a spiritual entity could achieve the limitlessness of interactive language, putting words together in indefinitely many ways,” as Jessica Riskin wrote in The Restless Clock (63).
Thus, resting beyond the scope of the mechanical philosophy and, therefore, the scientific boundaries drawn by the Cartesians, human language could only be accounted for with appeal to an immaterial soul from which this ability springs.
The Twentieth Century Reformulation
In the early twentieth century, Descartes’ original problem was effectively reformulated and split into two, though not to much fanfare.
The eventual establishment of general computability theory (then-recursive function theory) meant, in fact, that the infinite generativity from a finite system is possible, to which the work of Turing, Church, Post, Gödel, and others each contributed. As Turing wrote in 1936: “It is possible to invent a single machine which can be used to compute any computable sequence” (241). Thus, in principle, one could conceive of a mechanism could generate infinite expressions from finite means.
Chomsky has for decades been clear about how this impacted his own linguistic work:
With these intellectual tools available, it becomes possible to formulate what we may call the Basic Property of human language. The language faculty of the human brain provides the means to construct a digitally infinite array of structured expressions; each is semantically interpreted as expressing a thought, and each can be externalized by some sensory modality, such as speech…When externalized, it can be used for social interactions, although this is only a first approximation to what may properly be called thought.
The original challenge posed by Descartes, Chomsky continues, is subject to new “qualifications” in light of this new intellectual tool:
We must distinguish the internalized system of knowledge from the processes that access it. The theory of computability enables us to establish the distinction…
Nevertheless, Chomsky writes,
There has been considerable progress in understanding the nature of the internal language, but its free creative use remains a mystery. (2)
And he concludes:
The origins of computational atoms remain a complete mystery. So does the Cartesian question of how language can be used in its normal creative way, in a manner appropriate to situations, but not caused by them, incited and inclined, but not compelled. The mystery holds for even the simplest forms of voluntary motion. (6)
The Cartesians had no conception of how a finite, physical object (the human body/brain) could yield the infinite diversity evidenced by ordinary and routine examples of human language use. The distinction between (material) body and (immaterial) soul was not grounded in the preferred explanatory framework of the mechanical philosophy, but instead was a startling exception to an otherwise - in their view - comprehensive account of the world. There was therefore no substantive distinction to be made between what today linguists refer to as linguistic competence and linguistic performance:
Competence: the abstract knowledge of language that underwrites, but is separate from, its use.
Performance: the recruitment of this knowledge to use language in arbitrary circumstances; normal usage.
The new tools of computability theory thus allowed, in part, the distinction between “internal” language (a system of knowledge) and its “externalizations” (e.g., speech).
Whereas before Descartes’ original remarks targeted a single problem of infinitely diverse human language, computability theory enabled its reformulation: the capacity for infinite generativity from a finite physical mechanism residing in the mind is now within the scope of scientific inquiry.3 Left aside, untouched, is the normal use of this language in arbitrary circumstances; the stimulus-free, unbounded, yet appropriate use of language.
At most, these new tools allow for the postulation of a generative procedure of some kind or another that enables the human mind to generate an infinite array of structured expressions; its use is another matter. (Notably, this problem of language use does not depend on whether this is a single mechanism or multiple, or whether it is domain-specific or domain-general.4)
For the sake of clarity and laziness, here is what I have said elsewhere:
We should be clear about what exactly this reformulation of the Cartesian problem into competence and performance entails, as it is crucial: computability theory is an intellectual tool that allows linguists to study the properties of a single system capable of producing an infinite array of structured linguistic expressions. This, in generative linguistics, is a system of internalized knowledge.
This intellectual tool, however, comes with a corollary: a system of internalized knowledge is distinct from the use of this system in production, i.e., the actual use of language. Computability theory thus provides for the distinction to be made between the internal competence and externalized performance. But it crucially has nothing to say about ordinary language use; the original Cartesian problem (see, Chomsky, 2017, 2). The innovation of the mid-twentieth century was to distinguish “the internalized system of knowledge from the processes that access it” (Chomsky, 2017, 2), thereby making a distinction between internal language and speech (or signs) that the Cartesians appear not to have made for they lacked the tools to make it.5
Misreading Historical Precedent
Current debates about whether LLMs replicate human language - or, whether human language is thinking or merely the expression of thought - largely neglect this history and the distinctions drawn with the new tools of the twentieth century. Indeed, a quick-and-dirty rip of Descartes’ remarks quoted above from their historical context are often provided as a way of saying: Descartes was clearly wrong, LLMs demonstrate this, and here’s how we can account for these new language machines.
The thing is: Descartes was wrong - sort of. Him and his Cartesian followers in the years that followed the Discourse on Method were ignorant of the possibility of infinite generativity from finite means. More than this, as may be more customary in the sciences today, to posit a dualism in which the human mind (or soul) occupies a distinct domain from the rest of the physical world is unnecessary. One of the implicit admissions of the competence-performance distinction is to say: we believe we can get our grips on the internal systems that underlie language use, but we do not believe the same of language use itself.
At the same time, once one steps back from LLM-hoopla, we see that Descartes’ remarks are typically read in the same way one reads a LinkedIn post: you sort of the get the gist, but all you really need to know is we’re dunking on the oldheads. Yet, in his criteria to distinguish humans from machines, Descartes clearly rules out what would today be referred to as either parroting words (the mere output of words) or stimulus-control (in contrast to the stimulus-freedom of human language use):
for we may easily conceive a machine to be so constructed that it emits vocables, and even that it emits some correspondent to the action upon it of external objects which cause a change in its organs…
Descartes suggests that a machine emitting words (vocables) is conceivable, i.e., possible with the mechanical framework, and thus not sufficient for humanity. He is, in the same breath, also saying that a machine might emit words as a consequence of some action from the outside (e.g., pushing a button, pulling a lever, etc.) which directly affects the configuration of the machine’s internal “organs” and leads to the emission of words. This, too, is conceivable, i.e., possible within the mechanical framework, and thus not sufficient for humanity.
Notice how Descartes framed his argument: there was no real distinction to be made within the mechanical philosophy between a machine that could output a limitless diversity of words and a machine that could select for its own purposes a limitless diversity of words; the distinction today between competence and performance. He was wrong, but interestingly so, in that the selection of words for use in an arbitrary circumstance thus far does not have a mechanical (or computational) solution, whereas the output of limitless diversity of words appears to stand in contrast.6
The same framing is employed in the quote from Cordemoy above:
But yet, when I shall see, that those Bodies shall make signes, that shall have no respect at all to the state they are in, nor to their conversation…I shall not be reasonable, If I believe not, that they are such, as I am. (18-19)
Cordemoy’s first indicator that a being under investigation possesses a mind/soul is that its use of language “shall have no respect at all to the state they are in, nor to their conversation,” itself an expression that human language use, being non-mechanical in nature, is unfixed to local stimuli yet appropriate to local stimuli.
Dead-Metaphor Machines
It is ironic that, today, those who see the most promise in LLMs as models or replications of (some aspects of) the human mind unknowingly adopt something closer to the original formulation of Descartes’ problem - with the twist that they believe LLMs to have solved the problem. Frequently, there is no need to distinguish between the capacity for language and its actual use; the entire “problem” of natural language, as it were, is the capacity. Having “solved” the problem, LLMs put Descartes in his rightful place. There is no need to posit a theoretical construct to account for the unexplained for there is nothing unexplained, currently or potentially; a kind of naive rendition of the mechanical philosophy.
The confidence of these claims is not unusual. Figures within intellectual movements at the peak of their rise will often reach a point where they feel comfortable dismissing alternative, usually pre-existing, positions. And the desire to secure this comfort is not to be underestimated: it hurts to be criticized. To reach a point at which one feels they can shield themselves - the identities they derive from their view of the world - from future criticism is effectively to reach a point where one can convince themselves the criticism is meaningless; it need not hurt because it is so deficient. Contrary claims become de-legitimized.
I focus on this because the reactions to the confidence of these claims carry their own risks, chief among them accepting the conception of “language” put forward by the most bullish voices and the negligence to the concept’s history. Indeed, a number of commentaries have emerged in recent months that, if I were to boil them down into their most basic points, take the following form:
Any system that exhibits the limitations A, B, and C in the domains of X, Y, and Z is not intelligent (in some meaningful sense of the word).
Language is insufficient to overcome the limitations of A, B, and C.
Large Language Models are models of language.
Therefore, Large Language Models are insufficient for intelligence (in some meaningful sense of the word).
Authors vary, but this captures a fair deal of LLM criticism today.
Eryk Salvaggio wrote a beautifully self-reflective essay on the topic in October. He invokes the first two premises above:
But I think one of the problems of AI is that any LLM’s “model of the human mind” is actually a model of language. The assumption that language can contain all possible phenomena is a bold claim. Language barely contains math.
…
Likewise, what we’re looking at with AI is not “how the human brain works” but approximating models of how isolated parts of some human brains work.7
Earlier, Salvaggio referenced the alleged link between language and thought:
Most of us produce language, and we assume others who produce language produce language in similar ways. When we assume language reflects thinking, we may also assume that all thinking reflects our thinking. This can lead us to the faulty conclusion that language reflects the journey of thought.
This gets closer to the third premise and conclusion above.
Benjamin Riley, author over at Cognitive Resonance (I recommend subscribing), wrote in The Verge this month an essay titled “Large Language Mistake.”
Riley outlines a position favorable to LLMs, attributable to the industry writ large:
The common feature cutting across chatbots…are that they are all primarily “large language models.” Fundamentally, they are based on gathering an extraordinary amount of linguistic data (much of it codified on the internet), finding correlations between words (more accurately, sub-words called “tokens”), and then predicting what output should follow given a particular prompt as input. For all the alleged complexity of generative AI, at their core they really are models of language.
He then invokes, contra this, cognitive and neuroscience:
The problem is that according to current neuroscience, human thinking is largely independent of human language — and we have little reason to believe ever more sophisticated modeling of language will create a form of intelligence that meets or surpasses our own. Humans use language to communicate the results of our capacity to reason, form abstractions, and make generalizations, or what we might call our intelligence. We use language to think, but that does not make language the same as thought. Understanding this distinction is the key to separating scientific fact from the speculative science fiction of AI-exuberant CEOs.
This is a full endorsement of the four-pronged argument above. Riley ends with a reference to an LLM as a “dead-metaphor machine.” (It has a ring to it!)
There’s much I agree with in both Salvaggio’s and Riley’s remarks.
More important than my agreement, I believe engagement with cognitive science (and philosophy of mind) is immensely valuable, even if one disagrees with the position chosen - as I do, in the remarks that follow. Such engagement is unavoidable when trying to wrap one’s head around LLMs, even if the disciplines are used only for the purpose of distinguishing biological from artificial phenomena (a valuable exercise in its own right). I also believe that, because LLMs exist and are widely diffused, the challenge to explain them and/or distinguish them from humans is widely felt, evidenced in no small part by the fact that people continue to argue about them, often without the terminologies of these disciplines, though firmly within their boundaries nonetheless.
Resisting on Shared Ground
All that said, I have two points of caution about using cognitive science and adjacent disciplines to argue against LLM-mania in light of what we saw above.
First, exactly what is language? I am not here to answer this, but notice that all engaged in this debate (save for select academics, where there is naturally more nuance) find themselves on shared ground: “language,” for them, is essentially what you see on this page; characters, whether in the form of text or speech (or sign).
Both Salvaggio and Riley seem to agree that the text-data (converted into tokens) on which LLMs are trained is language. Moreover, one of the motivations for certain post-training techniques used for “reasoning”-based models, summed up nicely here, echoes this: the pre-training dataset for LLMs are products of human reasoning. Yet, this dataset is incomplete. LLMs should be trained on the processes that lead to them - internal verbalizations made explicit, and so forth. These ‘derivational traces’ are thus included in post-training. (It’s quite beautiful, even if its relation to human reasoning is somewhat illusory.)
This is an odd point of agreement, as it is not at all clear that language is what you are reading right now; or, at least, that a full account of language would include only its various modalities. Indeed, the point Chomsky made above was that the new intellectual tools of computability theory meant that new distinctions could be drawn. Among them, the distinction between “internal” language as a system of knowledge and “external” language as the externalizations of this system of knowledge through various modalities like speech, text, etc.8
Second, some of the research to which Riley points is quite representative of a particular point of view, though its authors neglect problems we have outlined here.
In particular, an article by Fedorenko et al. in Nature from 2024 titled, “Language is primarily a tool for communication rather than thought.” The title alludes to a debate within the field: whether language is primarily a tool for human thought or primarily a tool for human communication. They argue the latter.
The piece is not about AI, but its implications would be clear. As Riley notes in The Verge:
Our cognition improves because of language — but it’s not created or defined by it.
Take away our ability to speak, and we can still think, reason, form beliefs, fall in love, and move about the world; our range of what we can experience and think about remains vast.
But take away language from a large language model, and you are left with literally nothing at all.
First, let me say that I recommend Fedorenko et al.’s article to those interested in the subject. They argue that language is designed for a communicative function; it transmits knowledge, thoughts, and feelings. They argue that language does not necessarily mediate thought (577-579), that simply being in possession of intact language capacity does not entail intact reasoning abilities (580), and that evidence in the domains of sounds, word forms, word meanings, syntax, and ambiguity together suggest that language is for efficient communication (580-582). They summarize: “Language is unlikely to be a critical substrate for any form of thought” (582).
Now, let me explain why I find this highly deficient.
Note immediately that Fedorenko et al. decline to invoke a competence-performance distinction. As a result, there is no distinction made between language as a capacity of the human mind/brain and ordinary language use. Indeed, left undistinguished, they assume that language has a function of some pre-defined kind (i.e., communication). There is no discussion of the Cartesian problem, including the many citations one could find on this topic from their primary target: Chomsky.
Yet, as we have seen, language use is not teleological. There is no meaningful sense in which the stimulus-free, unbounded, yet appropriate use of language to express the contents of one’s mind can be reduced to a single “function” like communication.
Additionally, the authors fall into a Skinner-esque trap. Skinner stretched the notion of stimulus-control in language use beyond its utility. Fedorenko et al. effectively void the scientific value of the concept communication by stretching it beyond its limit, neglecting one of the most basic Cartesian insights: language use is stimulus-free; individuals may not use language at all or they may use it for anything.
The language-for-communication thesis has nothing whatsoever to tell us about this. It merely glosses the unfixed and unbounded uses of language as “communicative,” voiding any precision or depth entailed by the concept of communication as it attempts to cover voluntary language use.
The point applies as well to AI, though one would miss its import if they assume language is merely the words on this page or dead characters in the wind. Language use appears to not be for anything prior to a person’s decision to use it. Language use is without pre-defined, pre-given ends. Language is, in fact, a ‘quite useless’ tool.
This would appear to have something to do with the concept of human intelligence, for what it’s worth. I have said what I can on that elsewhere.
Illusions of Understanding
One thing I have cautioned against in reference to scholars who argue that LLMs are models of human language is the illusion of progress in linguistic science, not realizing the depth and qualities of the problems faced.
The point applies as well to (industry) critics: the grounds on which these debates are held often share assumptions that should be, at least, questioned. There is no obligation to change one’s views on these matters. There is no need for or expectation of consensus. Some problems, like voluntarily language use, may remain unexplained.
Language is bizarre. LLMs are a contender for the most fascinating scientific artefact of this century so far.9 Neither should be given short shrift. These challenges will be with us long after the current batch of CEOs.
There were other points of focus, but language is the clearest.
Novel either to the individual or in human history outright.
An enduring confusion about generative linguistics is that Universal Grammar is an explanation for the infinite generativity of sentences from finite means. The infinite generativity in question here is not of sentences, but of the internal mechanism that enables an unbounded range of novel linguistic behaviors. This persists in part because sociohistorical conventions like English, Mandarin, French, etc. can also exhibit an indefinite range (by embedding clauses within one another to make an indefinitely long sentence, which you can do in some languages). But the question is not what gives sentences this unbounded range; it is what enables novel behavior, whether that behavior leads to this or not. Descartes could posit an immaterial soul, but had no basis within the mechanical philosophy that would allow him to posit any more than this to make sense of unbounded linguistic behavior.
Though, I argue in a forthcoming chapter that the problem is best accounted for by postulating a modular and generative language faculty, quite minimal in what it enables, though not aiming to explain creative language use and instead only what makes it possible.
Chomsky has apparently felt that his use of the term “production” is widely misread, and should not be conflated with language use in ordinary circumstances (as if to say, internal language is used in production). I have occasionally used the term to mean exactly this.
This should not be read as me agreeing that LLMs have mastered natural language.
Another passage from the same piece is nonetheless worth noting here: “It also points to the weird ideology of AI’s push to “general intelligence,” which eschews cross-attentional models or “switching between specifically trained models” in favor of building one big universalized latent space to do everything. That, even if it worked, would also not be “how humans think,” so it is a weird way to go about it.”
I rather like this diagram from a recent Elliot Murphy article, here on page 2.
I suspect more ordinary technologies, like renewable energy tech, will be more impactful this century than any tech resembling LLMs. But that’s another story.