From Random Weights to Local Inference

The easiest way to train a Transformer is to hide most of the process behind a framework. The easiest things to miss are how text becomes tokens, how gradients change random weights, how checkpoints become a reloadable model, and why excellent validation loss can coexist with poor answers to real questions.

I built Local Transformer Training Lab to run the entire chain on a personal computer:

Dataset generation
→ Tokenizer
→ Decoder-only Transformer
→ Pretraining
→ Supervised fine-tuning
→ Independent evaluation
→ Checkpoints and export
→ Local inference

The experiments ran on a MacBook Air with 16GB of memory, using Apple MLX and the Metal GPU. Every model began with random weights. None was a fine-tune of Qwen, Llama, or another pretrained model.

Why train from scratch?

Model APIs are excellent for building products, but they hide several questions that matter when a system fails:

• How does tokenization change sequence length and output safety?
• What does a causal attention mask actually look like in code?
• Why do pretraining and SFT need different data and loss regions?
• Why can a model with low validation loss fail after a simple paraphrase?
• Where should repetition, invalid text, and empty answers be handled?
• What must be saved to reload a model reliably in another process?

The value of training from scratch is not the parameter count. It is that no anomaly can be dismissed as “the black box.” The relationship between data, architecture, optimization, and inference becomes concrete.

Generation zero: make the complete loop work

The first programmer-world-tiny dataset contained roughly 10 MiB of locally generated material covering Go, Linux, Redis, MySQL, Docker, code completion, and engineering interviews.

The model was intentionally small: four decoder blocks, four attention heads, a hidden width of 128, and an FFN width of 384. The 128-token version had 742,272 parameters; extending context to 512 tokens increased it to 791,424.

The implementation in src/model.py contains the complete path:

Token embedding + position embedding
→ RMSNorm
→ causal multi-head self-attention
→ residual connection
→ RMSNorm
→ GELU feed-forward network
→ residual connection
→ language-model head

The causal mask prevents position t from seeing future tokens. Next-token training shifts one sequence by a single position: the model receives every token except the last and predicts every token except the first.

Training uses AdamW, gradient clipping, periodic validation, and separate best, checkpoint, and final exports. A reloadable package contains three essential files:

config.json
model.safetensors
tokenizer.json

Weights without configuration cannot reconstruct the network. A model without the exact tokenizer cannot interpret its token IDs.

Byte tokenization: simple, universal, and expensive

The first tokenizer represented UTF-8 bytes directly. Its vocabulary contained 263 entries: 256 byte values plus special markers. It required no tokenizer training, was reversible, and covered Chinese, English, and code.

The cost was sequence length. A Chinese character usually consumes three byte tokens. A nominal 512-token context therefore holds far fewer than 512 Chinese characters. A model can also generate an incomplete UTF-8 byte sequence, producing the replacement character � at the terminal.

After 2,500 steps, the 512-token model reached a best validation loss of 0.135433 and a final perplexity of 1.1514 in about 110.63 seconds. Those numbers looked impressive. Its real answers did not.

The train and validation records came from the same template distribution. The low loss primarily proved that the model had learned those regularities—not that it possessed general programming knowledge.

Generation one: BPE, two training phases, and answer-only loss

v1 expanded the model to 8,534,272 parameters: eight layers, eight heads, a hidden width of 256, and a 4,096-token ByteLevel BPE vocabulary.

The efficiency difference was immediate:

“Why is Redis so fast?” (Chinese prompt)
Byte tokenizer: 28 tokens
BPE tokenizer: 7 tokens

The more important change was separating training into two phases.

Domain pretraining

Pretraining material was transformed into technical notes and complete code rather than question-answer wrappers. The objective was to learn domain language, code structure, and next-token regularities.

Supervised fine-tuning

SFT data removed synthetic phrases such as “additional constraint” and “simulated business context,” then applied semantic deduplication and category balancing. Loss was computed only after the first <|assistant|> marker, so the model optimized the answer rather than spending most of its capacity reproducing user text and control tokens.

v1 completed 800 pretraining steps and 1,500 SFT steps. Its best SFT validation loss was 0.235517. Independent paraphrased prompts exposed a different reality: Redis, MySQL, and Docker answers could bleed into one another; identity answers picked up interview templates; paraphrases generalized poorly; and some prompts produced an immediate end marker.

The useful conclusion was not merely that loss improved:

Scale, BPE, and training technique improve representation and fit. Natural variation, supervision quality, and independent evaluation determine whether the model can actually answer.

Generation two: guarantee valid output first

ByteLevel BPE still allowed v1 to produce byte fragments that did not form valid UTF-8. Instead of blindly scaling the model again, v2 switched to a Unicode character tokenizer.

The corpus contained 750 distinct characters plus nine special tokens, for a vocabulary of 759. Every emitted token represents a valid Unicode code point. Unknown characters map explicitly to <|unk|>, and generation no longer produces �.

v2 has 6,825,728 parameters, eight layers, eight heads, and a 512-token context. It completed 800 pretraining steps and 1,500 SFT steps in 115.45 seconds.

Character tokenization is not simply “better” than BPE. Sequences are longer, and without a KV cache generation is slower. The experiment separated three objectives that are often collapsed into one metric: model capability, tokenizer efficiency, and output validity.

Inference is a product layer

A script that can load weights is not yet a reliable inference interface. v2 added:

• a repetition penalty over the most recent 64 tokens;
• stopping after four identical consecutive tokens;
• stopping after the same trigram repeats three times;
• truncation when control markers leak into the answer;
• removal of Unicode replacement characters;
• an explicit fallback for empty answers;
• normalization of common short prompts;
• --no-history to prevent untrained multi-turn context from compounding errors.

Disabling history by default may be the least intuitive decision. A 512-token window does not mean the model learned multi-turn dialogue. Without reliable multi-turn SFT, feeding one wrong answer into the next prompt makes errors compound. When a capability is not ready, doing less is more trustworthy than pretending to support it.

Evaluating a teaching model honestly

A random split of one template distribution produces optimistic metrics. The project eventually used three layers of checks:

1. Implementation correctness: tokenizer round trips, mask behavior, shapes, save/reload, and minimal training tests.
2. In-distribution metrics: training loss, validation loss, perplexity, and gradient norm.
3. Behavior outside the template: human paraphrases, domain leakage, empty answers, repetition, invalid text, and multi-turn contamination.

The most valuable artifacts are often the failures, not the best number. Each failure points toward the layer that needs work: data, tokenizer, loss, architecture, or inference policy.

Five lessons I am keeping

1. Data structure matters more than raw size

Ten MiB of highly templated data can produce a low loss and weak generalization. Deduplication must go beyond string equality and prevent semantic templates from leaking across train and validation sets.

2. The tokenizer is part of the architecture

It determines vocabulary parameters, sequence length, context utilization, unknown-character behavior, and output validity. It is not a replaceable preprocessing detail.

3. Pretraining and SFT solve different problems

Pretraining learns language and domain distributions. SFT teaches the model how to respond within a task protocol. Mixing both into one undifferentiated next-token dataset obscures those roles.

4. Loss answers only the question it was designed to answer

In-distribution validation loss asks whether the model predicts similar tokens. It does not ask whether the model understands natural user language. Metrics must correspond to the capability being claimed.

5. Inference controls do not hide model weakness

Stopping repetition, falling back safely, and disabling history do not make the model smarter. They make its boundary honest, stable, and observable. Reliable systems must limit how errors propagate, not only maximize correct outputs.

What comes next

The project now completes the journey from random weights to local inference, but it remains a teaching model. The next valuable steps are not more synthetic templates or blindly increasing training steps. They are:

• licensed, source-traceable Chinese technical and code data;
• 1,000–5,000 human-written questions, paraphrases, and follow-ups;
• semantic topic isolation across train, validation, and test;
• compiler tests for code and rule-based or human scoring for technical answers;
• controlled comparisons across 10M, 30M, and 50M parameters;
• a KV cache and explicit speed-memory-quality evaluation.

Training from scratch did not make modern language models look less impressive. It made the hidden coordination behind every good answer visible: data, tokenization, architecture, optimization, evaluation, and inference policy all have to work together.

Repository and full experiment reports: hh696-wq/local-transformer-training-lab