Atlas
kb://library/byte-and-tokenizer-free-modelsstable2026-06-17

Byte- & tokenizer-free models — deleting the vocabulary and reading raw bytes

libraryeducationtokenizationbyte-leveltokenizer-freebyt5megabytebltbyte-latent-transformercaninecharformercharacter-levelrobustnessmultilingual

Byte- & tokenizer-free models — deleting the vocabulary and reading raw bytes

The other half of the [[tokenization]] story. That article explained the hand-built, non-neural function that turns text into ~100k integers — and catalogued its failure modes (can't spell strawberry, botches arithmetic, melts down on SolidGoldMagikarp, treats some languages as 4× more expensive). This article is the complement: what if we delete the tokenizer entirely and feed the model raw bytes? Then every glitch token, every fixed-vocab boundary, every <UNK> simply cannot exist. The catch is the one the tokenization article already named — byte sequences are much longer, and attention is quadratic. The whole research lineage here is a fight against that length cost: ByT5 (just eat it), Charformer/CANINE (learn a soft downsample), MEGABYTE (fixed patches + a hierarchy), and the Byte Latent Transformer (BLT, Meta 2024 — dynamic entropy-based patches). Sourced from the primary papers; numbers attributed inline.

This is the direct counterpart to [[tokenization]] (the thing being deleted). Related: [[embeddings]] (what the integer IDs index into), [[attention-and-transformers]] and [[flash-attention]] (why length is the central cost), [[long-context]] (the same quadratic enemy), and [[scaling-laws]] (the framing for "is this a better use of compute?").

What it is (intuition first)

A standard LLM never sees text. A non-neural [[tokenization]] step chops the string into subword tokens, looks up an integer ID for each, and the model works on those IDs. The vocabulary (~50k–200k entries) is frozen at training time and learned by a separate algorithm (usually byte-pair encoding) that gradient descent never touches.

A byte- or tokenizer-free model throws that step away. The input alphabet is just the 256 possible byte values (or the ~1,000 Unicode codepoints, depending on the design). There is no learned vocabulary, no merge table, no <UNK>, no "this word wasn't in training." encode is literally str.encode("utf-8"). Every string — any language, any emoji, any typo, any made-up product name, any code — is representable exactly, because bytes are the universal representation of text.

The intuition for why you'd want this: the tokenizer is a brittle, frozen, hand-tuned layer bolted onto an otherwise end-to-end-learned system, and the [[tokenization]] article shows how many "the model is dumb" moments trace straight back to it. Deleting it removes a whole class of bugs at the root.

The intuition for why it's hard was already stated in that same article's list of "obvious extremes, all bad": one token per byte means enormous sequence length. English averages roughly 4 bytes per subword token, so a byte model's sequences are ~4× longer than a BPE model's for the same text. Attention is O(n²) in sequence length ([[attention-and-transformers]]), so naively a byte model costs ~16× more attention compute and burns context budget fast. The entire field below is variations on one question: how do you get the robustness of bytes without paying the full length tax?

Why it matters

The failure modes the [[tokenization]] article documents are not edge cases — they are structural, and bytes dissolve most of them:

  • Spelling & character tasks. "How many r's in strawberry?" is hard for a BPE model because strawberry may be 2–3 opaque tokens with no visible letters. A byte/char model sees every letter and can count. ByT5 was shown to be substantially more robust to noise and on spelling/pronunciation-style tasks than its subword sibling mT5 (ByT5, arXiv:2105.13626).
  • Multilingual fairness. BPE vocabularies are dominated by high-resource languages, so the same sentence can cost 3–5× more tokens (= more money, less context) in, say, Burmese than in English. Bytes are a flat, uniform cost: UTF-8 charges by content, not by who was in the training corpus. (Caveat below — bytes are not perfectly uniform either.)
  • No glitch tokens. SolidGoldMagikarp-style tokens — IDs that exist in the vocab but were almost never trained, producing bizarre behaviour — cannot exist when there is no vocab. The frozen-vocab attack surface is gone.
  • No fixed-vocabulary lock-in. A subword vocab is chosen once and welded in; a new domain or language you under-tokenize is stuck that way for the model's life. Byte models have nothing to lock in.
  • Robustness to noise & adversarial input. Typos, casing, unicode tricks, and byte-level perturbations degrade subword models sharply (one wrong character can re-segment a whole word into different tokens). Byte models degrade gracefully because a one-byte change is a one-byte change.

The cost side is honest and singular: longer sequences. Everything below is an engineering answer to that.

How it works (the lineage of fixes)

Four families, roughly in order of how cleverly they fight the length tax.

1. ByT5 — just eat the length (2021)

The simplest tokenizer-free design: take T5, replace the SentencePiece tokenizer with raw UTF-8 bytes (vocab = 256 + a few special IDs), and train. No clever pooling — the transformer operates directly on byte sequences (ByT5, arXiv:2105.13626).

To partly offset the length, ByT5 rebalances the parameter budget: because byte sequences are long, it makes the encoder much deeper than the decoder (heavy encoder, light decoder) rather than the symmetric T5 split. Findings: ByT5 is competitive with mT5 on standard benchmarks, wins on noisy data and character-level tasks, and is more parameter-efficient on the embedding side (no giant vocab embedding matrix). The honest cost the paper reports: byte models are slower (longer sequences = more compute), especially for long inputs and at inference. ByT5 is the "pay the tax, gain the robustness" baseline everything else tries to beat.

2. Charformer & CANINE — learn a soft downsample (2021)

If raw bytes are too long, learn to compress them before the expensive transformer stack — but learn it end-to-end instead of hand-building a vocab.

  • CANINE (arXiv:2103.06874) operates on Unicode codepoints with no explicit vocabulary, using a hash-based embedding (so any codepoint maps to a vector without a lookup table) and a strided convolution to downsample the long character sequence to a shorter one before the deep transformer, then up-samples for token-level tasks. Tokenization-free, and competitive with / better than mBERT on multilingual benchmarks at comparable cost.
  • Charformer (arXiv:2106.12672) introduces GBST (Gradient-Based Subword Tokenization) — a soft, differentiable module that scores candidate character n-gram blocks and learns a downsampled "latent subword" representation as part of the model. The segmentation is learned by gradient descent, not by a separate BPE pass. Charformer matched or beat subword baselines while being faster than naive byte models.

The shared idea: replace the frozen, non-neural tokenizer with a learned, differentiable downsampler. The limitation: these still use fixed-stride / fixed-rate pooling — compression is roughly uniform across the input, which is wasteful (some spans are predictable and could be compressed hard; others shouldn't be).

3. MEGABYTE — fixed patches + a hierarchy (2023)

MEGABYTE (arXiv:2305.07185) attacks length with a two-level hierarchy and fixed-size patches:

  1. Split the byte sequence into patches of a fixed size P (e.g. P = 8 bytes).
  2. A large global transformer runs over the patches — so its sequence length is n/P, slashing the quadratic cost by ~P².
  3. A small local transformer runs within each patch to predict its individual bytes.

Because the expensive global model only sees n/P positions, MEGABYTE makes modelling million-byte sequences tractable and decodes faster (much of the work is parallel within patches). It showed byte-level modelling could be compute-competitive with subword transformers at scale — the key proof that "bytes are too long" is an engineering problem, not a law of nature. The weakness it bequeaths to BLT: patches are a fixed size, so a patch boundary falls in the middle of structure regardless of content, spending equal compute on a trivial run of spaces and a dense burst of code.

4. Byte Latent Transformer (BLT) — dynamic entropy patches (Meta, 2024)

BLT (arXiv:2412.09871) is the current state of the art and the most-cited recent result. Its insight: don't patch by a fixed size — patch by how *surprising* the bytes are.

A small auxiliary byte-level language model estimates the entropy (next-byte uncertainty) at each position. BLT then groups bytes into patches so that low-entropy (predictable) regions become long patches and high-entropy (surprising) regions become short patches — i.e. compute is allocated where information is. A long predictable run (e.g. the tail of a common word) is one cheap patch; a hard, information-dense boundary gets fine-grained attention.

Architecturally BLT is three pieces: a lightweight local encoder (bytes → patch representations), a large latent global transformer (operates over the patches — this is where most parameters and FLOPs live), and a lightweight local decoder (patches → bytes). The headline claims from the paper:

  • BLT matches the performance of a tokenizer-based Llama 3 model trained at the same scale (up to 8B params / ~4T training bytes) while using up to ~50% fewer inference FLOPs — because dynamic patching averages larger patches than a fixed tokenizer's tokens (BLT, arXiv:2412.09871).
  • It demonstrates a new scaling axis: with no fixed vocabulary you can simultaneously grow patch size and model size under a fixed inference budget — a knob tokenizer models don't have.
  • It inherits the byte-level robustness wins (noise, spelling, low-resource languages, character manipulation) over the BPE baseline.

The conceptual punchline: a frozen BPE tokenizer is a crude, content-blind, static form of "patching." BLT makes patching learned, dynamic, and compute-aware — which is exactly the move [[tokenization]] flagged as the missing piece.

Key ideas & tradeoffs

  • The single tradeoff: robustness/no-vocab vs. sequence length. Bytes buy you universality and kill a whole bug class; they cost you longer sequences. Every architecture above is a different bet on how to recover the length.
  • Static vs. learned vs. dynamic segmentation. BPE = static, non-neural, content-blind. Charformer/CANINE = learned but ~uniform-rate. MEGABYTE = fixed-size patches. BLT = dynamic, entropy-driven patches. The trend line is "let the model decide where the boundaries go, by content."
  • Hierarchy is the workhorse trick. MEGABYTE and BLT both put the big transformer over coarse units (patches) and a small model over fine units (bytes). This is the same instinct as [[mixture-of-depths]] — spend compute unevenly — applied to the input granularity.
  • Embedding-matrix savings are real but modest. No 100k×d_model vocab embedding matrix is a genuine parameter saving (ByT5 noted this), but it's a small slice of a large model's budget; the robustness and no-vocab wins are the real motivation, not parameter count.
  • It's a clean fit for multilingual & code. Domains where BPE's frozen, English-skewed vocabulary hurts most (low-resource languages, dense code, novel symbols) are exactly where byte models shine.

Honest caveats & open questions

  • Byte uniformity is not perfect fairness. UTF-8 encodes ASCII in 1 byte but many scripts (Cyrillic, Arabic, CJK, Devanagari) in 2–4 bytes per character, so byte sequences for those languages are still longer than for English. Bytes remove the vocabulary bias but not the encoding length asymmetry — the multilingual-fairness claim is "much better," not "solved."
  • The length tax is reduced, not abolished. Even BLT's ~50% inference-FLOP claim is a specific comparison at a specific scale (≤8B, ~4T bytes) on the paper's chosen evals; it is not a universal "byte models are cheaper" law. Read it as "competitive at this scale," and treat extrapolation to much larger scales as open. (Numbers attributed to arXiv:2412.09871.)
  • Tooling & ecosystem inertia. The entire serving stack — KV-cache sizing, context-window accounting, billing-per-token, speculative decoding, grammar constraints — is built around subword tokens. Byte models are not a drop-in; much infrastructure assumes a tokenizer.
  • BLT's entropy model is an extra moving part. The auxiliary byte-LM that decides patch boundaries is itself trained and adds complexity; how robust the patching is under distribution shift (e.g. a language the entropy model saw little of) is a fair open question.
  • Adoption is still early. As of this writing the dominant frontier models remain tokenizer-based. ByT5/CANINE/Charformer/MEGABYTE/BLT prove the approach works and is competitive, but "tokenizer-free wins at the frontier" is a direction, not yet a settled outcome — state it as promising, not decided.
  • Don't overclaim the death of BPE. The honest framing (and the one [[tokenization]] takes) is that tokenization is a brittle, deletable layer and a deeply entrenched, well-optimized one. Byte models are the principled long-term answer; whether they displace BPE soon is genuinely uncertain.

How it connects to OpenAlice

  • Complement to the tokenization article. [[tokenization]] is "here's the brittle layer and its bugs"; this is "here's how the field is trying to delete it." Read together they're the full story of how text enters a model. The glitch-token / spelling / multilingual-cost failures catalogued there are the motivation for everything here.
  • Multilingual robustness matters for Alice. Alice converses in Russian and other non-English languages (e.g. helping with a Russian-language diploma). The BPE multilingual-cost penalty is a direct product concern — non-English chats silently cost more tokens and less effective context. Byte/BLT-style models are the structural fix; worth tracking for any future base-model decision.
  • Robustness for agentic/code work. OpenAlice's autonomous-coding direction (repo→PR, SWE-bench) leans on exact handling of code, identifiers, and whitespace — precisely where subword tokenization is leakiest and byte-level exactness helps. A relevant axis when evaluating base models for the coding loop.
  • The "spend compute where information is" theme recurs. BLT's entropy-based dynamic patching is the same family of idea as [[mixture-of-depths]] and the test-time-compute work — allocate compute non-uniformly by difficulty. Useful mental model across the library, not just for input encoding.
  • No production wiring implied. This is a research-tracking article. Switching Alice's base model to a byte-level architecture is not a current task; the value here is understanding the tradeoff before any such decision.

Sources