When a model can write code, review a contract, explain quantum mechanics, and discuss whether it is conscious, we instinctively imagine a mind behind the screen.
Yet when we open the system and follow the computation downward, we do not find desire, experience, or intention. We find tokens, vectors, parameters, attention weights, and repeated matrix multiplication.
This creates an uncomfortable question:
If a large language model is fundamentally predicting the next token, does it understand anything—or is it merely excellent at producing answers that look like understanding?
My original position was blunt: modern language models are not real intelligence. They predict the next character using probability. More parameters, more layers, and attention still reduce to matrix operations; the program itself understands nothing.
That intuition captures an essential technical fact, but it is too absolute. My revised position is:
Large language models are artificial intelligence and exhibit a form of functional understanding. But there is no convincing evidence that they possess human subjective consciousness, a persistent self, or intrinsic intention. Complex internal representations are not the same as living in and experiencing the world.
The model predicts tokens, not characters
Modern language models generally predict the next token, not the next character. A token may be a Chinese character, part of an English word, punctuation, a number, or a fragment of code.
Autoregressive generation can be summarized as:
existing context
→ tokenization
→ embeddings
→ many Transformer layers
→ probability distribution over the vocabulary
→ select one token
→ append it to the context
→ repeat
Given “Paris is the capital of,” the model does not simply retrieve the next word from a record. It calculates a probability for every candidate token. Sampling rules then select the actual continuation.
The GPT-4 technical report explicitly describes GPT-4 as a Transformer-style model pre-trained to predict the next token in a document. That is not an insult. It is the core training objective.
At the bottom, it really is matrix computation
At the physical execution level, language-model inference consists largely of:
- vector and matrix multiplication;
- similarity calculations inside attention;
- softmax normalization;
- feed-forward networks;
- nonlinear activations;
- residual connections and normalization.
Self-attention projects each token representation into Query, Key, and Value vectors:
Q = XWq
K = XWk
V = XWv
Attention(Q, K, V)
= softmax(QKᵀ / √d) V
The computation estimates which positions in the context matter for the current representation and how much information should be combined from them.
“Attention” is a mathematical name, not evidence that the model focuses as a person does. Artificial “neurons” are computational units inspired by biology, not equivalents of brain cells.
After dozens or hundreds of layers, the model produces probabilities for the next token. The GPU is indeed executing linear algebra at enormous scale.
But “it is only math” does not settle the question
It is tempting to stop here and conclude that the model understands nothing. The problem is that a low-level description does not automatically negate a high-level capability.
A music file is binary data. A computer is transistor state. Human neural activity can be described through electrochemical events. We do not conclude that melody or human understanding is unreal merely because the substrate follows physical rules.
The better question is not whether the substrate is mathematics. It is:
What internal organization emerges from the computation, and what can the resulting system do reliably?
This is what the “stochastic autocomplete” description often leaves out.
Prediction forces the model to learn structure
A tiny model trained on a few sentences may only memorize local correlations. Predicting the next token across a vast and diverse corpus is much harder.
To reduce prediction error, a model must capture some combination of:
- grammar and reference;
- relationships among entities, properties, and events;
- document structure and style;
- syntax, types, and dependencies in code;
- patterns linking problems to solutions;
- world knowledge and causal regularities reflected in language.
To complete “The glass fell from the table onto concrete, so it probably…,” a useful model benefits from representing that glass is brittle, gravity moves objects downward, and impact can cause breaking. Those ideas may not be stored as human-readable propositions, but distributed representations can support the prediction.
“It predicts the next token” and “it learns complex representations” are therefore compatible. The prediction task is difficult enough to force compression of many structures expressed through language.
Scale is not the whole explanation
Model capability comes from more than parameters and depth.
| Factor | Contribution |
|---|---|
| Parameter scale | Capacity to store and combine patterns |
| Depth | Multiple stages of transformation and abstraction |
| Attention | Context-dependent information composition |
| Training data | The language, knowledge, and biases available to learn |
| Objective | What behavior training rewards |
| Post-training | Instruction following, preferences, safety, and tool behavior |
| Inference-time compute | Search, verification, and additional reasoning steps |
| External tools | Current facts, calculation, memory, and action |
Parameters are not database rows. A fact or capability is usually distributed across many weights, and one weight participates in many patterns.
Calling every advance “more GPUs and parameters” is as reductive as calling the model a digital brain.
“Understanding” hides three different questions
Disagreement persists because people use one word for different phenomena.
1. Statistical and structural understanding
Can the system identify relationships and regularities in language and other representations?
Modern models clearly do this well. They recognize semantic similarity, transform expression, follow code dependencies, and extract contractual structure.
2. Functional understanding
Can the system apply concepts to new tasks, explain relationships, revise a plan after feedback, and complete work that historically required cognition?
In many domains, models demonstrate some functional understanding. A model may enter an unfamiliar repository, locate a defect, and propose a valid patch. Its probabilistic substrate does not erase the functional result. Weather forecasting is probabilistic too, yet it can represent something real and useful.
3. Subjective understanding
Does the system know that it is reasoning? Does it have experience, intention, desire, a stable self, or a first-person point of view?
Here, we have no reliable evidence that language models understand as humans do.
A model saying “I am afraid of being shut down” does not prove the experience of fear. The statement may simply be probable in context. A model can describe pain precisely without injury, nerves, or the experience of suffering.
The careful conclusion is:
Language models display structural understanding and some functional understanding, but have not been shown to possess subjective understanding.
Form is not the same as meaning
Bender and Koller distinguish linguistic form from meaning. Text training exposes a model to forms and their relationships. Human meaning also involves connections among language, communicative intent, shared environments, and lived experience.
A person understands “fire is hot” not only by reading the sentence but perhaps by approaching heat, feeling danger, receiving warnings, and acting differently in the world.
A text-only model has no such life history. Multimodal training exposes models to images, sound, and video. Robotics and tools provide limited environmental feedback. But sensor input alone does not establish human embodiment or subjective experience.
This helps explain why a model can offer a profound explanation and then fail on a trivial variation. Its representations can be broad and powerful without being consistently grounded in the world.
Modern AI systems do more than pre-training
“A next-token predictor” accurately describes the foundation, but not the entire modern system. Current systems can also include:
- supervised fine-tuning and preference optimization;
- reinforcement learning and verifiable rewards;
- inference-time search and comparison;
- retrieval-augmented generation;
- code execution, calculators, and business tools;
- multimodal input and environmental feedback;
- agent state, planning, approval, and outcome evaluation.
These additions usually do not turn the model into an intentional subject. They provide a probabilistic model with external memory, sensors, actuators, and verification.
A model does not develop a desire to email a customer because an email tool exists. The objective comes from people, system instructions, or reward. Permission comes from policy. Correctness still depends on evidence and feedback.
Why does it hallucinate so confidently?
Next-token training explains a characteristic failure: the model is optimized first to produce a plausible continuation, not to guarantee that every sentence corresponds to a verified fact.
When knowledge is missing, continuing to generate often remains more natural than remaining silent. A model may invent a paper, API, or event in fluent language.
Reliable production systems therefore need more than eloquence:
model proposes a judgment
→ retrieval or tools obtain facts
→ structured constraints
→ rules and evaluations verify
→ people approve high-risk actions
Treating the model as a probabilistic reasoning component is more realistic than treating it as an omniscient digital employee.
Is it “real AI”?
If artificial intelligence means machines completing tasks normally associated with human intelligence, language models clearly qualify. Claiming that they are “not AI” is difficult to defend technically.
But if real intelligence requires a persistent self, embodied experience, self-generated goals, intrinsic concern for consequences, and subjective consciousness, current language models are far from demonstrated human-like intelligence.
The issue is not a binary choice between fully intelligent and entirely unintelligent. Intelligence may be multidimensional: language, prediction, planning, learning, embodiment, social relationship, selfhood, and consciousness are not the same faculty.
Language models are extremely strong along some dimensions. Along others, convincing evidence is absent.
My final position
I would no longer say, “It is only probability, so it understands nothing.” The word “only” hides the complexity that large-scale prediction can produce.
I would also not say that a model understands the world as a person does. Fluent language invites us to project a subject, emotions, and intentions onto software without sufficient evidence.
My position is:
A large language model is a high-dimensional probabilistic system trained through enormous amounts of data and computation. Its core objective is token prediction, and its substrate is matrix computation. In pursuing that objective, it develops internal representations that support language, knowledge, reasoning, and tool use. It can exhibit functional understanding, but has not been shown to possess human subjective consciousness, a stable self, or intrinsic intention.
This does not diminish its usefulness. An airplane need not flap like a bird to fly. A machine may not need to reproduce the human brain to perform valuable intelligent work.
But we must know what the system is in order to decide how much to trust it, which authority to grant it, and which responsibilities must remain human.
The most interesting question may no longer be whether it resembles us. It may be:
When a probabilistic system without demonstrated consciousness can perform an expanding range of intelligent tasks, do we need to redefine intelligence itself?