llama2.c — Llama-2 Inference in One C File

"My fun weekend hack: llama2.c. Lets you train a baby Llama 2 model in PyTorch, then inference it with one 500-line file with no dependencies, in pure C." — Andrej Karpathy, July 2023

If you have ever wondered what actually happens when an LLM generates a token — not the metaphors, the arithmetic — llama2.c is the single best place to look. It is the entire forward pass of a Llama-2 transformer, written in plain C, with zero dependencies beyond the standard library and libm. You can read it end to end in an afternoon, and when you do, the mystery of "inference" collapses into a pile of for loops and matmul.

This article is a guided tour of that file: what it is, why it matters, and a real walkthrough of the mechanics, line by line where it counts.

What it is

llama2.c is Andrej Karpathy's minimal, self-contained implementation of the Llama-2 transformer, split into two halves:

• `train.py` — a PyTorch training script (a lightly modified [[nanogpt]]) that trains a small Llama-2 model from scratch.
• `run.c` — a single C file (~700 lines in the canonical telling; closer to ~950 lines in the current master once sampling, tokenizer, and CLI plumbing are counted) that loads the trained weights and runs inference — the autoregressive generation loop — with no external dependencies.

The flagship demo is a 15-million-parameter model trained on the TinyStories dataset (synthetic children's stories with a tiny vocabulary). It is small enough to generate coherent text at ~100 tokens/sec on a laptop CPU, yet it uses the exact same architecture as Meta's 7B Llama-2 — RoPE, RMSNorm, SwiGLU, optional grouped-query attention. The neural net is identical; only the dimensions differ.

The project is MIT-licensed (© 2023 Andrej Karpathy). It began as a weekend fork of nanoGPT, retuned from the GPT-2 architecture to the Llama-2 architecture. Per Karpathy's own README, "the only notable changes from GPT-1/2 architecture is that Llama uses RoPE relatively positional embeddings instead of absolute/learned positional embeddings, a bit more fancy SwiGLU non-linearity in the MLP, RMSNorm instead of LayerNorm, bias=False on all Linear layers, and is optionally multiquery."

Quick start:

git clone https://github.com/karpathy/llama2.c
cd llama2.c
wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin
make run
./run stories15M.bin

Three pretrained TinyStories checkpoints live on HuggingFace (karpathy/tinyllamas): stories15M (~60 MB, 256 context), stories42M and stories110M (~400 MB, 1024 context).

Why it matters

Real production inference engines — llama.cpp, vLLM, TensorRT-LLM — are sprawling codebases full of CUDA kernels, paged attention, speculative decoding, and quantization schemes. They are fast but opaque. You cannot learn how a transformer runs by reading vLLM any more than you can learn anatomy from a running marathoner.

llama2.c matters because it is the dissection table. It strips inference down to its irreducible core:

1. It demystifies "inference." A forward pass is embed → (for each layer: attention + FFN) → norm → classify. That's it. Everything else (batching, KV paging, kernel fusion) is optimization layered on top of this skeleton.
2. It is the canonical reference for the Llama family. Dozens of ports exist — llama2.java, llama2.rs, llama2.go, llama3.c — all tracing back to this file because it is small enough to fully understand and re-implement.
3. It is a teaching artifact. It sits in the same lineage as Karpathy's [[nanogpt]], [[nn-zero-to-hero]], and the broader [[llm-from-scratch]] / [[microgpt-build-an-llm-from-scratch]] curriculum — the "build it yourself to understand it" school.
4. It shows the architecture lineage. Reading it next to [[attention-and-transformers]], [[positional-encoding]], and [[quantization]] turns abstract diagrams into running code you can printf-debug.

It is the practical companion to the related Atlas note on [[llm-c]] — the broader "LLMs implemented in C" tradition that llama2.c essentially founded.

How it works (real mechanics + code walkthrough)

The four structs

Everything in run.c revolves around four structs. Understand these and you understand the program.

`Config` — the architecture hyperparameters, read straight from the checkpoint header:

typedef struct {
    int dim;           // transformer dimension (the residual stream width)
    int hidden_dim;    // FFN inner dimension
    int n_layers;      // number of transformer blocks
    int n_heads;       // number of query heads
    int n_kv_heads;    // number of key/value heads (< n_heads ⇒ grouped-query)
    int vocab_size;    // tokenizer vocabulary size
    int seq_len;       // max context length
} Config;

`TransformerWeights` — pointers into the memory-mapped weight blob. Note there is no bias anywhere (Llama design choice):

• token_embedding_table (vocab_size, dim) — the lookup table that turns a token id into a vector.
• rms_att_weight, rms_ffn_weight, rms_final_weight — the per-dimension scale vectors for RMSNorm (one set per layer, plus a final one).
• wq, wk, wv, wo — the attention query/key/value/output projection matrices.
• w1, w2, w3 — the FFN matrices. In SwiGLU terms: w1 = gate, w3 = up, w2 = down.
• wcls — the final classifier that maps the residual stream to vocab-size logits (often tied to / shared with the embedding table).

`RunState` — the scratch buffers, allocated once and reused every step. This is where the activations live:

float *x, *xb, *xb2;             // residual stream + two norm/attn scratch buffers
float *hb, *hb2;                 // FFN hidden buffers (hidden_dim wide)
float *q, *k, *v;                // current token's query/key/value
float *att;                      // attention scores: (n_heads, seq_len)
float *logits;                   // output distribution over vocab
float *key_cache, *value_cache;  // THE KV CACHE

The key_cache and value_cache are the single most important optimization in the file — they store every past token's keys and values so attention never recomputes them. More on that below.

Loading the model — mmap instead of malloc

The checkpoint is not read into a buffer. It is memory-mapped:

void read_checkpoint(char* checkpoint, Config* config, ...) {
    FILE *file = fopen(checkpoint, "rb");
    fread(config, sizeof(Config), 1, file);   // header = the Config struct
    fseek(file, 0, SEEK_END);
    *file_size = ftell(file);
    fclose(file);
    *fd = open(checkpoint, O_RDONLY);
    *data = mmap(NULL, *file_size, PROT_READ, MAP_PRIVATE, *fd, 0);
    // weights are just VIEWS into the mapped file:
    memory_map_weights(weights, config, weights_ptr, shared_weights);
}

memory_map_weights then walks a single pointer through the blob, handing each weight matrix its slice:

w->token_embedding_table = ptr; ptr += p->vocab_size * p->dim;
w->rms_att_weight        = ptr; ptr += n_layers * p->dim;
// ...sequential pointer arithmetic for every weight tensor...

This is elegant: the OS pages in weights lazily, multiple processes can share one read-only copy, and there is no load-time deserialization. The .bin format is simply the `Config` header followed by every weight tensor concatenated in fp32. A 7B model → a 26 GB file (4 bytes × parameters).

The forward pass — one token, one position

forward(transformer, token, pos) is the heart of everything. It takes the current token id and its pos (sequence position) and returns the logits for the next token.

Step 0 — embed. Copy the token's row out of the embedding table into the residual stream x:

float* content_row = w->token_embedding_table + token * dim;
memcpy(x, content_row, dim * sizeof(*x));

Then loop over all n_layers transformer blocks. Each block has two sub-blocks: attention and FFN, each wrapped in an RMSNorm and added back as a residual.

Step 1 — RMSNorm + QKV projection. First normalize, then project into query/key/value:

rmsnorm(s->xb, x, w->rms_att_weight + l*dim, dim);
matmul(s->q, s->xb, w->wq + l*dim*dim, dim, dim);
matmul(s->k, s->xb, w->wk + l*dim*kv_dim, dim, kv_dim);
matmul(s->v, s->xb, w->wv + l*dim*kv_dim, dim, kv_dim);

RMSNorm itself is wonderfully simple — no mean subtraction (unlike LayerNorm), just normalize by the root-mean-square and scale:

void rmsnorm(float* o, float* x, float* weight, int size) {
    float ss = 0.0f;
    for (int j = 0; j < size; j++) ss += x[j] * x[j];
    ss = 1.0f / sqrtf(ss / size + 1e-5f);
    for (int j = 0; j < size; j++) o[j] = weight[j] * (ss * x[j]);
}

Step 2 — RoPE: rotate q and k. This is where position enters the model. Rotary Position Embedding pairs up adjacent dimensions and rotates each pair by an angle proportional to pos × frequency. Lower dimensions rotate fast, higher dimensions rotate slowly:

for (int i = 0; i < dim; i += 2) {
    int head_dim = i % head_size;
    float freq = 1.0f / powf(10000.0f, head_dim / (float)head_size);
    float val = pos * freq;
    float fcr = cosf(val);
    float fci = sinf(val);
    int rotn = i < kv_dim ? 2 : 1;   // 2 = rotate q AND k, 1 = q only (GQA)
    for (int v = 0; v < rotn; v++) {
        float* vec = v == 0 ? s->q : s->k;
        float v0 = vec[i];
        float v1 = vec[i+1];
        vec[i]   = v0 * fcr - v1 * fci;   // standard 2D rotation
        vec[i+1] = v0 * fci + v1 * fcr;
    }
}

The beauty of RoPE — covered in depth in [[positional-encoding]] — is that the relative position between two tokens is preserved purely through the geometry of these rotations. No learned position table, works for any length. The rotn trick handles grouped-query attention: when there are fewer KV heads than query heads, only the query dimensions beyond kv_dim need rotating.

Step 3 — write to the KV cache. The freshly-projected k and v for this token at this position are stored so future tokens can attend to them. loff is the per-layer offset into the cache:

int loff = l * p->seq_len * kv_dim;   // layer offset into the cache
// (k and v for this position are copied into key_cache/value_cache at loff + pos*kv_dim)

Step 4 — attention. For each head, compute the dot product of the current query against every cached key up to `pos`, scale by 1/sqrt(head_size), softmax, then take the weighted sum of the cached values:

#pragma omp parallel for private(h)
for (h = 0; h < p->n_heads; h++) {
    float* q = s->q + h * head_size;
    float* att = s->att + h * p->seq_len;
    // attention scores for this head over all positions 0..pos
    for (int t = 0; t <= pos; t++) {
        float* k = s->key_cache + loff + t * kv_dim + (h / kv_mul) * head_size;
        float score = 0.0f;
        for (int i = 0; i < head_size; i++) score += q[i] * k[i];
        score /= sqrtf(head_size);
        att[t] = score;
    }
    softmax(att, pos + 1);            // normalize scores into weights
    // weighted sum of values → xb
    for (int t = 0; t <= pos; t++) {
        float* v = s->value_cache + loff + t * kv_dim + (h / kv_mul) * head_size;
        for (int i = 0; i < head_size; i++) xb[i] += att[t] * v[i];
    }
}

The (h / kv_mul) indexing — where kv_mul = n_heads / n_kv_heads — is grouped-query attention in action: several query heads share one KV head via integer division. If n_kv_heads == n_heads, kv_mul == 1 and it degrades to ordinary multi-head attention. This single division is the entire GQA mechanism. (See [[attention-and-transformers]] for why this is the standard memory-saving trick for the KV cache.)

Then project the attention output back and add the residual:

matmul(s->xb2, s->xb, w->wo + l*dim*dim, dim, dim);
for (int i = 0; i < dim; i++) x[i] += s->xb2[i];   // residual connection

Step 5 — the FFN (SwiGLU). Normalize again, then run the gated feed-forward network. SwiGLU computes w2( silu(w1·x) ⊙ (w3·x) ):

rmsnorm(s->xb, x, w->rms_ffn_weight + l*dim, dim);
matmul(s->hb,  s->xb, w->w1 + l*dim*hidden_dim, dim, hidden_dim);  // gate
matmul(s->hb2, s->xb, w->w3 + l*dim*hidden_dim, dim, hidden_dim);  // up
for (int i = 0; i < hidden_dim; i++) {
    float val = s->hb[i];
    val *= (1.0f / (1.0f + expf(-val)));   // SiLU / swish = x·sigmoid(x)
    val *= s->hb2[i];                       // gate ⊙ up
    s->hb[i] = val;
}
matmul(s->xb, s->hb, w->w2 + l*dim*hidden_dim, hidden_dim, dim);   // down
for (int i = 0; i < dim; i++) x[i] += s->xb[i];   // residual

Step 6 — final norm + classifier. After all layers, normalize one last time and project to vocabulary logits:

rmsnorm(x, x, w->rms_final_weight, dim);
matmul(s->logits, x, w->wcls, p->dim, p->vocab_size);
return s->logits;

matmul — the one function that eats all the FLOPs

Every projection above is a call to one tiny function. Note that "matmul" here is really matrix-vector multiply — one token at a time:

void matmul(float* xout, float* x, float* w, int n, int d) {
    int i;
    #pragma omp parallel for private(i)
    for (i = 0; i < d; i++) {
        float val = 0.0f;
        for (int j = 0; j < n; j++) val += w[i * n + j] * x[j];
        xout[i] = val;
    }
}

That #pragma omp parallel for is the entire multi-threading story — split the output rows across cores. 95%+ of inference time lives in this loop. Everything else (RoPE, softmax, RMSNorm) is noise by comparison.

Sampling — turning logits into a token

run.c ships three strategies:

// greedy: always pick the most likely token
int sample_argmax(float* probabilities, int n) {
    int max_i = 0;
    for (int i = 1; i < n; i++)
        if (probabilities[i] > probabilities[max_i]) max_i = i;
    return max_i;
}

// multinomial: walk the CDF until a random coin lands
int sample_mult(float* probabilities, int n, float coin) {
    float cdf = 0.0f;
    for (int i = 0; i < n; i++) {
        cdf += probabilities[i];
        if (coin < cdf) return i;
    }
    return n - 1;
}

The third, nucleus / top-p (sample_topp), sorts probabilities descending and keeps only the smallest set whose cumulative mass exceeds p, then samples from that truncated set — this is what stops the model from "going off the rails" by occasionally picking a wildly improbable token. Temperature is applied to the logits before softmax (lower = sharper/greedier). Karpathy's recommended setting: -t 1.0 -p 0.9 (temperature 1.0 with top-p 0.9, both defaults).

The generation loop

The outer loop ties it together. It "force-feeds" the prompt tokens, then switches to sampling:

int token = prompt_tokens[0];
int pos = 0;
while (pos < steps) {
    float* logits = forward(transformer, token, pos);
    if (pos < num_prompt_tokens - 1)
        next = prompt_tokens[pos + 1];   // still inside the prompt → force it
    else
        next = sample(sampler, logits);  // past the prompt → sample
    pos++;
    if (next == 1) break;                // BOS token terminates
    char* piece = decode(tokenizer, token, next);
    safe_printf(piece);
    token = next;                        // feed output back as next input
}

The tokenizer is a from-scratch byte-pair encoding (BPE): encode prepends a dummy space, splits the prompt into bytes, then greedily merges the highest-scoring adjacent token pair (using merge scores from tokenizer.bin) until no merge improves things, falling back to byte-level tokens for unknown codepoints. See [[tokenization]] for the full theory.

Building it

gcc -O3 -o run run.c -lm            # plain
make runfast                         # -Ofast
make runomp                          # OpenMP multi-threaded
# clang -Ofast -fopenmp -march=native run.c -lm -o run
OMP_NUM_THREADS=4 ./run stories15M.bin -t 1.0 -p 0.9 -i "Once upon a time"

Key ideas & tradeoffs

• The whole transformer is ~6 operations. Embed, RMSNorm, matmul (QKV), RoPE, attention, SwiGLU FFN, classify. Memorize those and you have memorized "how an LLM runs."
• The KV cache is the real engine of efficiency. Without it, generating token N would recompute attention over all N previous tokens from scratch — O(N²) per step. The cache makes each step O(N), at the cost of memory that grows linearly with context. This tradeoff (compute vs. memory) is the central tension in all of modern LLM serving.
• `mmap` over `malloc`. Lazy paging, zero-copy, shareable across processes. A clean systems-level trick that real engines also use.
• OpenMP is the only concurrency. One #pragma over the matmul rows. No threadpools, no async — and it scales surprisingly far on multi-core CPUs.
• fp32 simplicity vs. int8 speed. run.c is pure fp32 for maximum clarity. Its sibling runq.c adds Q8_0 int8 quantization (symmetric, range [-127, 127], weights-only with fp32 norms, activations dynamically quantized at runtime) for ~3–4× speedup and ~4× smaller files. Export with python export.py model_q80.bin --version 2 --meta-llama .... This is the practical bridge to the deeper material in [[quantization]].
• Architecture, not scale, is what's shared. stories15M and Llama-2-7B run through the exact same forward(). Only the Config numbers change. That's the lesson: scale is a knob, not a different machine.

Honest caveats

Being honest about what llama2.c is not:

• It is not fast, and is not meant to be. It is a reference / educational implementation. For real workloads use llama.cpp, vLLM, or TensorRT-LLM. Karpathy's own numbers: stories15M ≈ 100 tok/s on an M1 Air; Llama-2-7B fp32 ≈ 4.6 tok/s on a 96-thread CPU box, ~14 tok/s with int8. That is a toy by production standards.
• fp32 caps you at ~7B. The README notes 13B+ models hit pointer-arithmetic overflow issues in the fp32 path — the indexing math wasn't written for tensors that large. 7B is the practical ceiling for the vanilla run.c.
• No GPU. It is CPU-only. There is no CUDA path in the canonical repo (third-party CUDA ports exist separately).
• Matrix-vector, not matrix-matrix. It processes exactly one token per forward() call — there is no batching. Real serving engines batch many sequences together for throughput; llama2.c deliberately does not, for clarity.
• No paged/optimized KV cache. The cache is a flat pre-allocated array sized to seq_len. No paging, eviction, or sharing à la vLLM's PagedAttention. Memory is reserved up front.
• Line count is "vibes," not exact. The famous "500-line" / "700-line" framing is from the original tweet and early README. The current master run.c is closer to ~950 lines once the BPE tokenizer, top-p sampler, chat mode, and CLI arg parsing are included. The spirit (one file, no deps) holds; the exact number drifted.
• TinyStories models are genuinely tiny. stories15M produces coherent toddler-grade stories, not a usable assistant. It is a demonstration of the architecture, not a capable model. Don't benchmark reasoning on it.
• Source caveat for this article: the run.c code excerpts here were extracted via automated fetch of the GitHub raw file and a markdown-conversion of the rendered repo page. The struct fields, RoPE loop, GQA indexing, and helper functions are quoted faithfully from those fetches and cross-checked against Karpathy's README prose. Treat the shape of each loop as authoritative; if you need byte-exact current code (variable names occasionally change between commits), read the file directly at the source URL. All fetches succeeded; no source was fabricated.

OpenAlice + Academy ladder

Where this sits in the Atlas knowledge library and the learning path:

Prerequisites — climb these first if the code above felt dense:

1. [[neural-network-from-scratch]] and [[nn-zero-to-hero]] — backprop and the basic building blocks.
2. [[math-for-ml-foundations]] — the linear algebra behind matmul and rotations.
3. [[attention-and-transformers]] — what Q/K/V attention is before you watch it run.
4. [[tokenization]] — what the BPE tokenizer in run.c is actually doing.

Core — `llama2.c` lives here, alongside:

• [[nanogpt]] and [[microgpt-build-an-llm-from-scratch]] — the training-side siblings; llama2.c's train.py is a nanoGPT derivative. Read nanoGPT for "how it learns," read llama2.c for "how it runs."
• [[positional-encoding]] — the deep dive on RoPE, the one rotation loop you saw above.
• [[llm-from-scratch]] — the full build-your-own-LLM arc that this file is the inference chapter of.
• [[llm-c]] — the broader "LLMs in C" tradition; llama2.c is its founding artifact.
• [[nanochat]] — Karpathy's later full-stack (train + chat) successor, for when you want the whole pipeline.

Next — once `run.c` makes sense, level up to:

• [[quantization]] — make it 4× smaller and faster (the runq.c Q8_0 path).
• [[flash-attention]] — how production engines make the attention loop you saw memory-efficient.
• [[deepseek-architecture]] and [[mixture-of-experts]] — what modern frontier models add on top of this Llama-2 skeleton.
• [[state-space-models]] — the main non-transformer alternative, for contrast.

Why OpenAlice cares. Alice's reasoning runs on hosted frontier models, not on a hand-rolled C engine — but understanding llama2.c is load-bearing for the team. It is the clearest mental model of what a "token" costs (every token = one full forward() = one pass through every weight), why the KV cache dominates context-length economics, and why quantization is the lever for cheap local inference. When we reason about latency budgets, context windows, or whether a small local model could handle a sub-task, the intuitions all trace back to the loops in this one file. It is the Rosetta Stone between "the model said X" and "here is the arithmetic that produced X."

Read the source — it really is one file: <https://github.com/karpathy/llama2.c/blob/master/run.c>. The fastest way to internalize this article is to clone the repo, download stories15M, and put a `printf` inside the attention loop.