The Moat in Large Language Models Is Not Just Code—It Is Data

Large language model code is no longer as mysterious as it was a few years ago.

Transformers are public. Attention is public. Mature open-source implementations now exist for pretraining, supervised fine-tuning, preference optimization, quantization, and serving. PyTorch, Hugging Face, DeepSpeed, Megatron-LM, TRL, and vLLM have turned many capabilities that once belonged only to large research labs into reusable engineering components.

For an experienced team, building a model that can train, fine-tune, and serve is genuinely easier than before.

But making a model run is not the same as making it useful.

My view has become increasingly clear:

As model architectures converge and training code becomes widely available, the real competition is shifting from “Can we implement the model?” to “Do we have the right data, how good is it, and can we continuously improve it?”

This does not make algorithms, compute, or systems engineering unimportant. Frontier training still demands sophisticated distributed systems, optimization, reliability, evaluation, and enormous compute. The difference is that baseline code is increasingly accessible, while valuable data and the system that produces it remain difficult to copy.

Parameters determine capacity; data determines what is learned

Machine learning has a simple and durable rule:

Garbage in, garbage out.

A language model does not first decide whether a document is true and then choose whether to learn from it. Pretraining still centers on predicting tokens from context. If a pattern appears often enough in the training data, the model may encode it in its parameters.

Fill the corpus with SEO-generated pages, and the model learns fluent emptiness. Fill a code corpus with duplicated repositories, obsolete dependencies, and incorrect implementations, and the model may produce unsafe code with greater confidence. If medical, legal, or financial material is wrong, adding parameters does not automatically turn it into truth.

“Data determines the ceiling” is directionally right, but it can be stated more precisely:

Parameter count determines how much pattern capacity a model has. Data determines which patterns it encounters. Algorithms and optimization determine how effectively those patterns are absorbed. Evaluation determines whether the team actually knows what the model learned.

Capability is not controlled by one variable. Architecture, compute, tokenization, objectives, data, post-training, and inference strategy interact. Data matters because it defines the boundary of experience available to the model.

Data quality is more than removing dirty text

Data quality is often reduced to deleting corrupted text, advertisements, explicit content, and duplicates. That is only the first layer.

For a language model, quality has at least five dimensions:

Dimension	Question
Correctness	Are facts reliable? Does the code compile and pass tests?
Diversity	Does the corpus cover different languages, domains, tasks, views, and styles?
Representativeness	Does its distribution resemble the situations the model will face?
Safety and compliance	Does it contain personal data, secrets, harmful material, or licensing risk?
Learnability	Is the content complete, clear, structured, and information-dense?

A cleanly formatted dataset without offensive language is not necessarily high quality. If every example repeats the same view, the resulting model will still be narrow and biased.

Quality and quantity are not simple opposites. General models need scale for coverage. Quality determines whether those tokens deserve the compute spent on them.

Do not invent percentages that were never disclosed

The precise composition of a training corpus is often among a model developer's most valuable secrets.

For Llama 3, Meta disclosed that the pretraining corpus contained more than 15 trillion tokens from publicly available sources. It was roughly seven times larger than Llama 2's corpus, contained four times as much code, and included more than 5% high-quality non-English data across over 30 languages. Meta also described heuristic filters, NSFW filters, semantic deduplication, quality classifiers, and experiments used to choose the final mixture.Meta's official Llama 3 announcement

Meta did not publish a complete mixture such as “50% Common Crawl, 17% GitHub, and 10% books and ArXiv.”

The rigorous claim is that large-model corpora commonly contain web pages, code, books, papers, encyclopedic material, conversations, and multilingual text. Their exact proportions depend on the target capabilities and should not be presented as official figures without a source.

That distinction reveals something important:

Architecture can be described in a paper. The data recipe often remains the commercial secret.

Phi-1 demonstrated data efficiency, not universal small-model superiority

Microsoft's Phi-1 is frequently cited as proof that data quality matters, but its conclusion is often overstated.

Phi-1 is a 1.3-billion-parameter model trained on approximately 6 billion tokens of filtered code data plus about 1 billion tokens of synthetic textbooks and exercises. It achieved strong results on coding benchmarks including HumanEval and MBPP.“Textbooks Are All You Need”

Its strongest lesson is:

In a focused domain with verifiable outcomes, information-dense data and a good teaching progression can dramatically improve training efficiency.

The work did not establish that a 1.3B model universally beats 7B models. Coding, world knowledge, multilingual capability, long-context behavior, and writing are different dimensions. One coding benchmark cannot stand in for all of them.

Data quality is not magic. It does not remove the boundary of the task or create knowledge absent from the corpus.

LIMA demonstrated “less but better,” not “1,000 equals 52,000”

LIMA fine-tuned an already pretrained 65B LLaMA model using only 1,000 carefully selected prompts and responses, and obtained strong instruction-following behavior.The LIMA paper

The precondition matters: the base model had already acquired substantial knowledge during pretraining. Supervised fine-tuning was largely teaching it how to organize, invoke, and express that capability.

The more defensible lesson is:

For a capable pretrained model, a small, carefully selected, well-covered instruction dataset can outperform a much larger collection of noisy and repetitive instructions.

It does not mean that any 1,000 examples equal 52,000 Alpaca examples, or that every enterprise needs only 1,000 labels. Base-model capability, task boundaries, coverage, and evaluation criteria all change the outcome.

What a production data pipeline should look like

Production data engineering is not a one-off cleaning script. It is a traceable, reproducible, measurable, and continuously evolving manufacturing system.

source registration and license review
        ↓
collection, parsing, and normalization
        ↓
main-content extraction and structure recovery
        ↓
language, domain, and content-type detection
        ↓
rule-based filtering and anomaly detection
        ↓
quality scoring and stratified sampling
        ↓
exact, fuzzy, and cross-source deduplication
        ↓
PII, secret, safety, and compliance processing
        ↓
train–evaluation contamination detection
        ↓
mixture design, curriculum, and token budgeting
        ↓
small-scale ablation experiments
        ↓
capability, safety, bias, and regression evaluation
        ↓
versioned release

The ordering is not universally fixed. Exact deduplication can move earlier to reduce downstream cost. Quality-aware deduplication may score first and retain the best document from each duplicate cluster. The best practice is not memorizing an order; it is giving every stage defined inputs, outputs, metrics, and lineage.

1. Govern the source before cleaning it

Every data batch should record its origin, acquisition date, license, usage restrictions, language, domain, and processing history.

Without provenance, a team cannot respond to copyright concerns, deletion requests, contamination, or reproducibility failures. What an enterprise needs is not merely object storage full of files, but a data catalog and lineage system.

2. Parsing quality constrains everything downstream

Menus, advertisements, recommendation links, cookie notices, and comments can overwhelm the main text of a webpage. PDFs introduce repeated headers, broken reading order, misplaced tables, and OCR errors.

If parsing destroys semantic structure, later stages are only cleaning characters in the wrong order. Data engineering begins by recovering document boundaries and structure as faithfully as possible.

3. Language identification needs more than fastText

fastText is a useful baseline, but short passages, dialects, low-resource languages, and code mixed with natural language are easy to misclassify.

Production pipelines usually combine model predictions, character-level rules, source metadata, and human sampling. Thresholds should be calibrated per language. Applying English-centric quality rules to every language often deletes the rarest multilingual data systematically.

4. Perplexity is not a synonym for quality

KenLM perplexity, heuristics, and learned classifiers are all useful, but no single score represents quality.

High perplexity may indicate corrupted text. It may also indicate specialized terminology, new knowledge, or a low-resource language. Aggressive global thresholds can remove the material that is hardest to replace.

A stronger approach combines multiple signals, buckets data by language and domain, performs human audits, and validates filtering choices through training experiments.

5. Deduplication is more than MinHash

One article may be syndicated dozens of times. A GitHub repository may have thousands of forks. A tutorial may be copied with only its title and a few sentences changed.

A complete strategy can include:

URL and content-hash deduplication;
exact document-level deduplication;
fuzzy deduplication using MinHash, LSH, or embeddings;
cross-source and cross-language deduplication;
contamination checks between training and evaluation sets.

The goal is not to delete the largest possible amount. It is to keep the most complete, trustworthy, and highest-quality member of each duplicate cluster.

6. PII and safety require more than regular expressions

Email addresses, phone numbers, identity numbers, home addresses, API keys, and internal code can all enter a corpus.

Regular expressions handle stable formats, but production systems also need entity recognition, secret scanners, classifiers, and human review. They need deletion workflows too: when data must be withdrawn, the team should know which dataset and model versions consumed it.

Harmful content should not simply be removed by keyword. A model needs some understanding of attacks, fraud, and dangerous material in order to detect and refuse them. Context, proportion, labeling, and the training objective matter more than mere occurrence.

A data mixture is a capability budget

Training compute is finite. Increasing the sampling rate of one source reduces the training opportunity available to another.

Code contains structure, long-range dependencies, and partially verifiable outcomes. More code may improve programming and some reasoning tasks, but it does not guarantee stronger general reasoning. Too much can crowd out natural language, domain knowledge, and multilingual ability.

More Chinese data generally improves Chinese understanding, expression, and knowledge coverage. The number of tokens required to encode Chinese, however, is governed mainly by the tokenizer's vocabulary and segmentation strategy. Corpus share and tokenization efficiency are related, but they are not the same variable.

Books and papers can provide dense knowledge and long-form structure. Social media can cover current language and public opinion while introducing misinformation, abuse, and demographic bias.

There is no universal golden ratio.

A data mixture is not a shopping list of ingredients. It is the product strategy projected into the training corpus.

A coding model, a Chinese legal model, and a general assistant should not share the same recipe.

The core of synthetic data is verification, not generation

As high-quality natural data becomes scarcer, synthetic data becomes increasingly useful. Strong models can generate textbooks, reasoning problems, code, question-answer pairs, and preference examples while controlling difficulty and covering rare tasks.

But synthetic data does not create trustworthy knowledge from nothing.

Teacher errors propagate to students. Repetitive templates reduce diversity. Repeatedly generating and filtering with the same model can amplify its existing biases.

The decisive capability is therefore verification:

check mathematics against answers and constraints;
compile code and run tests and static analysis;
execute tool calls and compare real outcomes;
ground factual questions in retrieved sources;
use multi-model review and human sampling where automatic verification is impossible.

Without a verifier, synthetic data is simply a cheaper and faster way to manufacture uncertainty.

Data engineering must close the loop with evaluation

After a cleaning run, the key question is not “How many tokens remain?” It is: which capabilities improved, which regressed, and why?

model failures and user feedback
        ↓
failure taxonomy and root-cause analysis
        ↓
identify gaps, bad samples, or mixture problems
        ↓
add, correct, resample, or relabel data
        ↓
small training runs and ablations
        ↓
offline evaluation, red teaming, and online observation
        ↓
promote into the next dataset version after quality gates

At least four kinds of metrics are required:

Data metrics: duplication, language distribution, domain coverage, quality scores, and PII detections;
Training metrics: loss by data bucket, gradient anomalies, token utilization, and convergence;
Capability metrics: target-task accuracy, code pass rate, factuality, and instruction following;
Risk metrics: hallucination, harmful output, privacy leakage, bias, and benchmark contamination.

Without versioned data, training cannot be reproduced. Without ablations, a team cannot prove that a data batch helped. Without production feedback, offline leaderboards cannot establish business value.

What the data moat means for an enterprise

Most companies should not pretrain a general foundation model from scratch. The better investment is converting internal knowledge, workflows, and expert feedback into governed and measurable data assets.

I would prioritize the following:

Select three valuable business tasks with measurable outcomes;
build a separate evaluation set for each task before collecting “all the data”;
inventory sources, permissions, licenses, sensitivity, and ownership;
create pipelines for cleaning, deduplication, redaction, versioning, and lineage;
validate value through RAG or small fine-tuning runs before scaling training;
continuously feed back corrections, reviews, and failure cases;
use business outcomes—not token counts—to decide where to invest.

The most defensible enterprise data is rarely another copy of the public web. It is authorized workflow data, expert judgment, customer feedback, failure cases, and outcome labels.

Such datasets may be small, but they are closely tied to real results and difficult for competitors to reproduce.

My conclusion

Algorithms and code still matter. Training systems, architecture, optimizers, compute, and inference engineering do not lose their value merely because open source exists.

But as the baseline stack becomes shared infrastructure, the advantages that remain hard to copy are:

long-lived, authorized domain data;
an understanding of provenance, bias, and quality;
expert judgment and feedback;
the ability to turn model failures into better training data;
evaluation tied to real business outcomes.

“Garbage in, garbage out” is only the first conclusion.

I would state the complete version this way:

Parameters determine capacity. Compute determines training scale. Algorithms determine learning efficiency. Data determines what the model ultimately learns.

As code becomes common infrastructure, the real competition is no longer about who can make a model run. It is about who can continuously feed it correct, scarce, lawful, and verifiable data—and prove that the model becomes better as a result.

That is the real data moat in the age of large language models.