Byte Latent Transformers and Tokenization-Free Language Models

Most language models today begin with a quiet compromise. Before the model ever sees your text, a separate tokenizer has already decided how to break it into pieces. That decision looks innocuous: split text into subword tokens, build a fixed vocabulary, and feed token IDs into a transformer.

The compromise is that this preprocessing step is external to the model. It bakes in assumptions about language, scripts, and even typos, long before any learning happens. As we push models into more languages and more precise tasks, that separation becomes a design constraint.

The Byte Latent Transformer (BLT) is one attempt to remove this boundary. Instead of accepting a fixed token vocabulary, it learns directly from raw bytes and builds its own internal grouping mechanism. The result is a tokenizer-free architecture that allocates compute where it is needed, rather than uniformly across a fixed set of tokens.

This article looks at why tokenization is a problem, how BLT restructures the pipeline, and what this suggests for the design of future language models.

Why Tokenization Became the Default

Tokenization originally solved a practical tension between two extremes:

• Character-level models treat every character as a unit. They preserve detail, but struggle to carry semantic structure. Long-range dependencies become expensive.
• Word-level models treat whole words as units. They carry rich semantics but break badly on rare words, new vocabulary, typos, and morphologically rich languages.

Subword tokenization sits between these extremes. Algorithms such as BPE (Byte Pair Encoding) and unigram tokenization build a vocabulary of frequent substrings. In practice this works well enough that most production LLMs are built around it.

However, the design has several persistent issues:

• Separate training pipeline. The tokenizer is trained independently from the model. Its vocabulary is frozen before model training and never revised in response to downstream behaviour.
• Language bias. A tokenizer trained mainly on English reflects English morphology. Underrepresented languages are split into long, awkward token sequences, inflating sequence length and degrading performance.
• Sensitivity to noise. Small typos or novel compositions can change token boundaries dramatically, producing representations the model has rarely seen.
• Uniform compute per token. The transformer spends the same amount of computation on a punctuation mark as on a technical term, because both appear as tokens in a flat sequence.

These problems are tolerable when models are overwhelmingly English-centric and compute budgets are high. They become more visible as we demand precise behaviour on multilingual and low-resource tasks, or try to optimise inference costs.

A Byte-Level Alternative

Byte-level models take a different starting point: the only primitive is the raw byte. There is no fixed vocabulary of subwords. Every input is a sequence drawn from the same small alphabet (typically 256 possible bytes).

On its own, this is not a complete solution. A naïve byte-level transformer would:

• Require long sequences to represent ordinary text.
• Struggle to infer higher-level structure from local byte patterns alone.
• Still pay uniform compute per byte, even when some regions are trivial.

BLT adds a second layer of structure on top of bytes: dynamically formed patches. Instead of making token boundaries once and for all in a preprocessing step, it lets the model learn where boundaries should lie, and how much context each boundary should cover.

From Bytes to Patches

BLT’s input pipeline has three distinct stages:

1. Byte embedding with local context.
2. Entropy-based patching.
3. **Patch-level representations for the main transformer.

Each stage exists to compensate for what would otherwise be a weakness of raw bytes.

Enriching Bytes with N-gram Context

A single byte has almost no semantic meaning. To make bytes useful units of computation, BLT augments each one with information about its immediate neighbours.

Conceptually, you can think of this as building small n-grams around each byte, then embedding them together. The model does not enumerate all possible byte combinations—that would explode the vocabulary. Instead, it uses hashing (for example, a rolling polynomial hash) to map local n-grams into a fixed-size embedding space.

The effect is that each byte carries a representation of both itself and its local context. This is still below the level of traditional subwords, but it is enough to distinguish, for example, a byte that appears inside a common word from the same byte inside a random identifier.

Entropy-Based Patching

The central idea in BLT is that not all parts of a sequence are equally difficult to predict. Predicting the next byte inside a repeated pattern is easy; predicting the first byte of a new technical term is hard. We would like the model to spend more computation on the latter than the former.

BLT uses entropy—a measure of uncertainty—to decide where to draw patch boundaries.

Roughly:

• It learns a local predictor over bytes.
• For each position, it estimates how uncertain it is about the next byte.
• It then segments the sequence into patches according to this uncertainty profile.

Two constraints govern how patches are formed:

• Global threshold. When the entropy of the next-byte prediction rises above a global threshold, BLT starts a new patch. These positions typically correspond to structurally important points: new words, unfamiliar character sequences, or shifts in style.
• Monotonicity constraint. When the change in entropy exceeds a relative threshold, BLT also starts a new patch. This captures surprising shifts in difficulty, even if absolute entropy remains moderate.

The resulting segmentation has an intuitive shape:

• Predictable regions (e.g., repeated characters, boilerplate, or simple formatting) become long patches.
• Unpredictable regions (e.g., rare words, multilingual fragments, or dense code) become short patches.

In other words, the sequence is compressed into variable-sized units whose length reflects how difficult the content is to model.

Local Encoder, Latent Transformer, Local Decoder

Once patches have been formed, BLT proceeds in three stages:

1. Local encoder.
2. Latent global transformer.
3. **Local decoder.

This architecture mirrors the intuition that:

• Local structure can be processed cheaply and in parallel.
• Global reasoning should operate on a compressed representation.

Local Encoder: From Bytes to Patch Representations

The local encoder takes the byte-level sequence, together with its n-gram embeddings and patch boundaries, and computes a representation for each patch.

This step has two main responsibilities:

• Aggregate byte-level information into a patch-level vector.
• Preserve enough detail that, later, the model can reconstruct bytes from patch representations.

Mechanically, this is implemented using lightweight transformer or attention layers operating within patches, followed by pooling or summarisation. The exact choice of pooling is a design decision, but the outcome is always a fixed-size vector per patch.

Latent Global Transformer: Reasoning Over Patches

The latent transformer is the core of BLT. It operates not on bytes, but on patch representations.

Its task is to:

• Capture long-range dependencies across patches.
• Predict the next patch representation in the sequence.

A useful way to read this design is that BLT does not model "next token" in the usual sense. Instead, it models next patch representation over a dynamically compressed timeline.

The benefits are straightforward:

• Fewer steps. Because predictable regions are merged into long patches, the transformer processes fewer positions overall.
• Targeted compute. Short patches appear exactly where the local predictor was uncertain, so the transformer spends more capacity on complex spans.

Empirically, this allows BLT to match the performance of strong token-based models while using substantially fewer floating-point operations at inference time.

Local Decoder: Back to Bytes

After the latent transformer has produced or updated patch representations, BLT needs to return to the byte domain. The local decoder handles this.

It takes as input:

• The predicted patch representations from the latent transformer.
• The local context and hidden states produced by the encoder.

From these, it reconstructs the original byte sequence, or generates new bytes when used generatively.

Conceptually, the local decoder reverses the work of the local encoder:

• It expands each patch representation back into a sequence of bytes.
• It uses cross-attention to combine global context (from the latent transformer) with local detail (from the encoder).

The key point is that both encoder and decoder are lightweight relative to the latent transformer. Most of the model’s capacity sits in the latent transformer, which now runs on a compressed sequence.

Compute Efficiency and Scaling Behaviour

The main claim of BLT is that, given similar parameter counts, it can match or approach the performance of strong token-based models while using less compute at inference time.

Several mechanisms contribute to this:

• Dynamic sequence length. Entropy-based patching reduces the number of positions the main transformer must process on predictable text.
• Adaptive granularity. The model naturally chooses finer-grained patches where semantics are dense and coarser patches where text is repetitive.
• Tokenizer-free input. There is no separate tokenization step to maintain or adapt. Training directly on bytes simplifies some aspects of the pipeline.

This leads to a new scaling axis. Token-based models typically scale along two dimensions:

• Model size (parameters).
• Data size (tokens).

BLT implicitly adds a third:

• Patch behaviour. By adjusting how aggressively patches are merged, one can trade off compute against fidelity without changing model parameters.

This is particularly relevant when deploying models under tight latency or energy budgets. Instead of distilling to a different architecture, you can control how much work the same architecture performs per input.

Multilingual and Low-Resource Considerations

Tokenizers trained on predominantly English corpora tend to fragment non-Latin scripts and low-resource languages into long token sequences. This has several downstream effects:

• Longer effective context lengths for the same textual content.
• Poorer representations for undertrained token patterns.
• Higher per-example compute for the same semantic content.

A byte-level model such as BLT does not make linguistic distinctions at the preprocessing stage. Every script is just bytes. The differences emerge from patterns learned during training, not from hard-coded token boundaries.

In practice, this can help on tasks that depend on:

• Orthography. Fine-grained knowledge of how words are spelled.
• Phonology proxies. Regularities in character combinations that reflect sound patterns.
• Low-resource translation. Languages with limited training data but consistent byte-level structure.

By design, BLT also avoids some failure modes of fixed tokenizers:

• Rare or newly coined words do not explode into long chains of unpredictable tokens; they are treated as unusual but still byte-composed spans.
• Small edits to text lead to small changes in the byte sequence, rather than large changes in token segmentation.

This does not make multilingual modelling trivial. It does, however, remove a class of artefacts introduced by tokenizer design.

Example: Conceptual Flow on a Short Input

To make the architecture concrete, imagine BLT processing the string:

"BLT models raw bytes."

At a very high level, the pipeline looks like this:

1. Bytes. The text is converted to its raw byte sequence.
2. Local context. Each byte is embedded together with hashes of its neighbouring n-grams.
3. Entropy profile. A small local model estimates the difficulty of predicting each next byte.
4. Patching. Entropy thresholds determine where patches begin and end: common patterns (like spaces and punctuation) might form long patches; unusual combinations (like new acronyms) short ones.
5. Local encoding. Each patch is summarised into a patch vector.
6. Latent transformer. These patch vectors flow through the main transformer layers, which reason about long-range structure.
7. Local decoding. Patch vectors and local context are combined to reconstruct or generate bytes.

This is deliberately abstracted away in typical usage. But from a design point of view, it highlights the division of labour: local modules focus on byte-level detail; the latent transformer focuses on global structure.

Design Implications

Looking at BLT as a piece of system design, several themes emerge.

Integrate Preprocessing Into the Model

Traditional tokenization is a fixed preprocessing step, conceptually outside the model. BLT moves this logic inside the learned architecture:

• Patching is driven by a learned entropy profile, not a static algorithm.
• Byte embeddings incorporate local context through trainable n-gram hashes.

This mirrors a familiar pattern in software engineering: push hard-coded heuristics into modules that can be tested, monitored, and evolved along with the rest of the system.

Align Compute with Uncertainty

Entropy-based patching is a concrete implementation of a general principle: spend computation where the model is uncertain.

Instead of:

• Fixing a uniform token size.
• Running the same amount of transformer compute everywhere.

we can:

• Use cheap local models to detect difficult regions.
• Allocate more expressive capacity to those regions via shorter patches and richer interactions.

This is analogous to profiling a program and optimising the hot paths, rather than optimising every line equally.

Treat Sequence Granularity as a Control Knob

Token-based models implicitly fix sequence granularity when they choose a tokenizer. BLT treats granularity as a dynamic property of the model’s state.

From an engineering perspective, this opens up new control surfaces:

• During training, you can experiment with different entropy thresholds to explore trade-offs between speed and accuracy.
• At inference, you can bias patching to be more conservative or aggressive depending on latency requirements.

Such knobs are valuable in production systems where workloads and budgets vary over time.

Avoid Overfitting to a Single Paradigm

BLT does not invalidate token-based models. It instead shows that there is viable structure below the token level:

• For tasks that depend heavily on exact spelling, multilingual robustness, or efficient inference, byte-level architectures can offer advantages.
• For broad conversational agents trained on massive corpora, subword tokenization and standard transformers remain strong baselines.

Good system design is rarely about replacing one pattern everywhere. It is about understanding where each pattern fits.

Reflecting on the Trade-offs

Moving from token-based to byte-level architectures like BLT involves clear trade-offs.

On the positive side:

• You eliminate a brittle, language-specific tokenizer from the pipeline.
• You gain a principled way to align compute with uncertainty via entropy-based patching.
• You unlock a new scaling axis by controlling patch behaviour rather than only model size.

On the challenging side:

• Training becomes more complex: the model must learn both local n-gram structure and global language structure from raw bytes.
• Tooling and ecosystem support for byte-level models are less mature than for token-based systems.
• Interpreting internal representations can be harder when everything is expressed at the byte level.

For beginners, the interesting question is not whether BLT replaces existing architectures, but where its properties match the problem at hand. If you are building systems that must operate across many scripts, tolerate noisy input, or run under tight inference budgets, it is worth considering architectures that remove the tokenizer as a fixed dependency.

BLT’s contribution is to show that such architectures can be competitive at scale. It reminds us that the choice of unit—character, byte, token, patch—is a design decision, not a law of nature. Treating that decision as a first-class part of the architecture opens up new ways to shape how our models think.