Train your own LLM from scratch — a hands-on path

For NAO + anyone who wants to *build*, not just read. Where [[microgpt-build-an-llm-from-scratch]] shows the algorithm in ~200 dependency-free lines, this is the practical workshop: a real (tiny) GPT in PyTorch you train on a laptop in under an hour. Grounded in `angelos-p/llm-from-scratch`↗ — a stripped-down, scaffolded re-derivation of Karpathy's nanoGPT, where you write every component (tokenizer → attention → trainer → sampler) and the docs explain the why behind each line.

What it is

A six-part, write-it-yourself workshop that produces a ~10.8M-parameter character-level GPT, trained on the ~1 MB Tiny-Shakespeare corpus, that runs to a coherent (Shakespeare-flavored) model in ~45 min on an M3 Pro Mac (also Colab / NVIDIA / CPU). It is explicitly a simplified [nanoGPT](https://github.com/karpathy/nanoGPT): nanoGPT's headline target is reproducing GPT-2 (124M) on OpenWebText (an 8×A100 node, ~4 days, val loss ~2.85); this strips that down to the essentials and to a model that fits a single laptop GPU. In fact the workshop's default "medium" config — n_layer=6, n_head=6, n_embd=384, block_size=256, vocab=65 — is byte-for-byte nanoGPT's own char-level Shakespeare demo config, which is the lineage made literal.

The build arc, end to end:

Repo shape: data/shakespeare.txt, a docs/ folder with the six tutorial markdowns, the model/train/generate Python you fill in, and pyproject.toml + uv.lock (installed via uv). Dependencies are deliberately tiny: PyTorch, NumPy, tqdm, and optionally tiktoken for the BPE detour. Device selection is automatic — MPS → CUDA → CPU — and the docs note MPS gives roughly 2–3× over CPU on M-series.

Why it matters

microGPT proves the math is small (4,192 params, hand-written scalar autograd). This proves the engineering is reachable: not a frontier run, but a genuine, working GPT you fully understand — built with the same primitives (PyTorch tensors, real autograd, AdamW) that production training uses. The value isn't the artifact (a Shakespeare babbler); it's that every knob you'll later tune on a real model — context length, head count, LR warmup, gradient clipping, temperature, top-k — you will have implemented by hand and watched move the loss curve. That is the difference between treating an LLM as a vending machine and treating it as a system you can reason about. It also closes the fundamentals trio: understand one model (microGPT) → build one (here) → fuse several ([[fusion-and-llm-councils]], [[mixture-of-agents]]).

How it works (real mechanics)

Part 1 — Tokenization: char-level, and why

"LLMs don't see text — they see sequences of integers." The tokenizer is the bijection. Char-level builds the vocab from the sorted set of unique characters:

chars = sorted(set(open("../data/shakespeare.txt").read()))
vocab_size = len(chars)                 # 65 for Tiny-Shakespeare
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
encode = lambda s:  [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)

The doc's central, non-obvious insight is a data-scale argument, not a quality law: on a ~1M-char corpus there are only ~4,225 possible character bigrams, so "every bigram appears many times" — dense statistics the model can actually learn from. Swap in GPT-2's BPE (50,257-token vocab via tiktoken.get_encoding("gpt2")) and most token pairs are too rare to estimate: the doc reports BPE training loss gets stuck around ~6.3 on Shakespeare, while char-level reaches ~1.5. Char-level's cost is ~3× longer sequences (you spend context budget on characters, not subwords) and a tiny vocab — but on small data that's the right trade. Vocab size also directly sizes the model: a 65×384 embedding table is ~25K params; GPT-2's 50,257×768 is ~19.3M. (rasbt's *LLMs from scratch*↗ goes the other way and builds a real BPE tokenizer — a good next step once you have more data.)

Part 2 — The Transformer: the forward pass, by hand

GPTConfig is a dataclass: vocab_size=65, block_size=256, n_layer=6, n_head=6, n_embd=384. The forward pass:

Embeddings. Two tables — token (wte, [65, 384]) and learned absolute positional (wpe, [256, 384]) — are summed element-wise. (Note: this is learned positions like GPT-2, not sinusoidal as in the original Transformer, and not RoPE as in modern LLaMA-class models — a real simplification worth knowing.) The token-embedding matrix is weight-tied to the output projection, so input and output token representations are forced consistent (a standard GPT-2 trick that also saves ~25K params).

Causal self-attention. CausalSelfAttention projects the input to Q, K, V with one linear, reshapes n_embd=384 into n_head=6 heads of 64 dims each, and computes scaled dot-product attention per head:

attn = softmax( (Q @ Kᵀ) / sqrt(head_dim) )   # head_dim = 64
out  = attn @ V

The 1/sqrt(head_dim) scale keeps the dot products from growing with dimension and saturating the softmax. Causality ("position i may only attend to 0..i") is enforced via is_causal=True — i.e. it rides PyTorch's fused scaled_dot_product_attention/FlashAttention path rather than materializing an explicit triangular mask, which is both faster and more memory-frugal. Heads run in parallel so different heads can specialize ("one head tracks which vowels follow consonants, another tracks line-break patterns"); outputs concatenate back to 384 and pass an output projection.

Feed-forward (MLP). Per block: Linear(384→1536) → GELU → Linear(1536→384) — the canonical 4× expansion. GELU (a smooth gate) is chosen over ReLU because "it doesn't have a hard cutoff at zero, which helps gradient flow."

Block = pre-norm + residual. Each of the 6 identical blocks is:

x = x + attn(LayerNorm(x))   # pre-norm: normalize the input to the sublayer
x = x + mlp(LayerNorm(x))    # residual: gradients flow straight through

Pre-norm (LayerNorm before the sublayer, GPT-2 style) stabilizes deep training versus the original post-norm Transformer; the residual highway lets gradients reach early layers undecayed. After 6 blocks: a final LayerNorm, then the tied linear head produces vocab_size=65 logits per position.

Where the ~10.8M params live: token emb ~25K (shared with head), position emb ~98K, and ~1.8M × 6 blocks ≈ 10.6M in the attention+MLP stack.

Part 3 — Training loop: making it learn stably

The objective is next-token prediction with cross-entropy — self-supervised, labels are just the input shifted by one. Batches are sampled by random offset:

ix = torch.randint(len(split_tokens) - block_size - 1, (batch_size,))
x  = torch.stack([split_tokens[i:i+block_size]       for i in ix])
y  = torch.stack([split_tokens[i+1:i+block_size+1]   for i in ix])   # x shifted by 1

Each of the batch_size=64 rows is a random block_size=256-char window; y is x advanced one step, so a single forward pass yields 256 next-char predictions per row (256×64 = 16,384 supervised targets per step — this density is why tiny models learn fast).

Optimizer: AdamW, lr=1e-3, weight_decay=0.01. The doc is refreshingly honest that it is deliberately simpler than production: "the full GPT-2 recipe (separate decay groups, betas=(0.9, 0.95), weight_decay=0.1) is designed for large-scale BPE training and is overkill here."

Gradient clipping by global norm: clip_grad_norm_(model.parameters(), max_norm=1.0) — caps the occasional spike before it "blows up the weights."

LR schedule = linear warmup then cosine decay:

# warmup_steps=100, max_lr=1e-3, min_lr=1e-4
if step < warmup_steps:
    lr = max_lr * step / warmup_steps
else:
    progress = (step - warmup_steps) / (max_steps - warmup_steps)
    lr = min_lr + 0.5 * (max_lr - min_lr) * (1 + cos(pi * progress))

Warmup "gives the optimizer time to calibrate its moment estimates before making large updates"; cosine decay = big steps early (explore), small steps late (refine).

Run shape: max_steps=5000, eval every 100 steps over 20 val batches, 90/10 train/val split, ~1.9 it/s on M3 Pro → ~45 min. The honest punchline: best val loss (~1.57–1.64) lands around step 1,500–2,500, after which the 10.8M model overfits a 1 MB corpus and starts memorizing — so the right move is early stopping, not running all 5,000 steps. That overfitting is itself a first-hand lesson in the data-vs-parameters balance.

Part 4 — Text generation: from predictor to writer

A next-token model becomes a writer via the autoregressive loop: predict, append, repeat — always cropping context to block_size:

for _ in range(max_new_tokens):
    idx_cond = idx[:, -block_size:]            # never exceed the context window
    logits, _ = model(idx_cond)
    logits = logits[:, -1, :] / temperature    # take last position, scale
    if top_k:                                  # keep only the top-k logits
        v, _ = torch.topk(logits, top_k)
        logits[logits < v[:, -1:]] = float("-inf")
    probs = torch.softmax(logits, dim=-1)
    idx = torch.cat([idx, torch.multinomial(probs, 1)], dim=1)

Two decode knobs, both acting on the logits before softmax:

• Temperature rescales logits (logits / T). T→0 → greedy/deterministic (and repetitive); T=1 → the model's raw distribution; T>1 flattens it, lifting rare tokens (creative but incoherent). The doc's sweet spot is T ≈ 0.7–0.9; it walks the spectrum explicitly (0.1 = boring, 0.8 = balanced, 1.5 = incoherent).
• Top-k masks all but the k highest logits to -inf, truncating the unreliable long tail before sampling. For the 65-char vocab, top_k≈40 is recommended.

Sampling is multinomial over the resulting distribution, not argmax: greedy decoding reinforces high-probability continuations into loops, whereas sampling "respects model confidence levels" while keeping variety. (A naive baseline loop in the docs uses argmax to show why you want sampling.)

Parts 5–6 — Experiment & scale

Part 5 wires it together: train on real data, read the loss curve, and try the BPE swap (where you watch loss stall — the Part-1 lesson made empirical). Part 6 scales model and data (Tiny 0.5M 2L/2H/128D ~5 min · Small 4M 4L/4H/256D ~20 min · Medium 10M 6L/6H/384D ~45 min) so you feel scaling laws on a laptop.

Key ideas & tradeoffs

• Char-level vs BPE is a data-scale decision, not a quality verdict. Small corpus → dense char statistics win; large corpus → BPE's shorter sequences and semantic units win. The ~6.3-vs-~1.5 loss gap on Shakespeare is the whole argument.
• The "real-trainer extras" are what make it *train*, not just *exist*. AdamW + grad-clip + warmup-cosine + pre-norm + residuals are precisely the pieces microGPT keeps minimal; they convert a bare algorithm into something numerically stable. This is the single highest-leverage thing the workshop adds over the algorithm-only path.
• Sampling quality is a runtime dial, not retraining. Temperature and top-k reshape output at inference — the same model is "boring" or "unhinged" depending on two scalars. This is exactly the lever an orchestrator tunes per call.
• Overfitting is a feature of the lesson. A 10.8M model memorizing 1 MB is the data-vs-params tradeoff you can watch on the val curve — early stopping beats more steps.
• Simplicity is chosen, and labeled. The docs repeatedly flag where they diverge from the GPT-2 recipe and why — a model for honest engineering writing.

Honest caveats & limitations

• It is a learning artifact, not a useful model. ~10.8M params on 1 MB produces grammatical-looking Shakespeare noise with no semantics, facts, or instruction-following. Nothing here transfers to "deploy a chatbot."
• Architecturally a 2019/2022-era GPT-2, not a 2026 LLM. Learned absolute positions (no RoPE), vanilla LayerNorm (no RMSNorm), GELU MLP (no SwiGLU/gated-MLP), dense attention (no GQA/MQA, no sliding-window), single-host fp32-ish training (no mixed precision, no FSDP/tensor-parallel, no flash-attn tuning beyond the fused SDPA call). Great for understanding; not the modern stack.
• Char-level caps the ceiling. It is the right call for this corpus but guarantees the model can't generalize like a subword model — don't read its weakness as a Transformer weakness.
• The trainer omits production must-haves. No checkpointing/resume detailed, no gradient accumulation, no distributed training, no EMA, minimal eval (val loss only — no perplexity-on-held-out-domain, no downstream task). Fine for a workshop; a real run needs all of these.
• Source-fidelity note: exact hyperparameter strings (e.g. top_k=40, warmup_steps=100, the cosine formula, the is_causal=True SDPA path) are read from the repo's docs/*.md via WebFetch on 2026-06-16; the fill-in Python (model.py/train.py/generate.py) is authored by the workshop participant, so line-level details may vary by your implementation. Treat the snippets here as faithful-to-the-docs, not as a verbatim single source file.

How it connects to OpenAlice

This article is the build rung of the fundamentals ladder, and it grounds several places where OpenAlice touches the same mechanics directly:

• Sampling knobs are live in the council. Temperature and top-k here are the same dials the [[fusion-and-llm-councils]] / [[mixture-of-agents]] panels set per-member — e.g. a higher-temperature "explorer" vs a low-temperature "verifier." Having implemented logits / T and top-k masking by hand is exactly the intuition needed to reason about why a council member produces diverse vs conservative drafts, and how [[model-routing]] should pick decode settings per task. NAO's note about the M11 council's "panel temperature" is this knob.
• Context-window cropping = our truncation discipline. The idx[:, -block_size:] crop is the toy version of the real budget problem [[mempalace]] / [[agent-memory-systems]] solve: a finite context means what you keep (recent vs retrieved-via-[[graphrag]]) decides what the model can condition on. Same constraint, bigger stakes.
• Training dynamics inform our eval bar. Loss curves, early-stopping on val loss, and the overfitting cliff are the literal levers behind any fine-tune or ranker work in alice-core's bench/lab rig — bench.blal.pro measures the same family of curves at the harness level.
• Atlas/graph parallel. [[graphify]] derives an AST call-graph; this article derives the other graph — the attention pattern over tokens. Both are "make the implicit structure explicit" — useful framing when explaining either to a newcomer.
• Library role. Sits beside [[microgpt-build-an-llm-from-scratch]] (the algorithm) and [[fusion-and-llm-councils]] (combining models), and is itself catalogued by [[llm-maintained-wiki]] — the meta-pattern this very file follows.

Takeaways

1. A working GPT is a laptop-hour away — the engineering is reachable, and you can write every line of it.
2. The "real trainer" extras (AdamW, grad-clip, warmup-cosine LR, pre-norm, residuals, top-k) are what turn the bare algorithm into something that trains stably and samples coherently.
3. Char-level vs BPE is a data-scale tradeoff (~1.5 vs ~6.3 loss on Shakespeare), not a quality law.
4. Overfitting on 1 MB is the lesson, not a bug — early-stop on val loss.
5. Understand → build → fuse: [[microgpt-build-an-llm-from-scratch]] → this → [[fusion-and-llm-councils]].

Sources: [`angelos-p/llm-from-scratch`](https://github.com/angelos-p/llm-from-scratch) (docs 01–04 read directly), Karpathy's [nanoGPT](https://github.com/karpathy/nanoGPT) (the upstream this simplifies), and rasbt's [LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) (the BPE/scaling-further sibling), cross-checked against the GPT-2 / Transformer lineage. Written via the [[deep-research]] method (M13). Deepened 2026-06-16 with exact hyperparameters, the attention/training/sampling mechanics, and an honest modern-stack-gap section.