Audio & Speech Models (ASR / TTS / Speech LLMs)

One sentence: This page is about teaching a computer to hear speech (ASR), to speak (TTS), and — at the 2024–2026 frontier — to do both inside a single transformer that listens and talks at the same time, by turning sound itself into a stream of discrete tokens that look, to the model, almost exactly like the word-tokens it already knows.

If you understand [[tokenization]] (text → integers) and [[multimodal-llms]] (pixels → tokens an LLM can read), you already hold the master key. Audio is the third modality that gets squeezed through the same doorway. The entire story below is one idea applied to sound waves: find a way to chop a continuous signal into a finite vocabulary, and the whole machinery of [[attention-and-transformers|transformers]] comes for free.

What it is (intuition first)

Sound, physically, is a wiggling air-pressure curve. A microphone samples that curve, typically 16,000 times per second for speech (16 kHz). So one second of audio is a list of 16,000 numbers. That is the rawest possible representation, and it is hostile to a transformer for two reasons:

1. It's continuous. There is no "vocabulary of sample values" to look up — the numbers are arbitrary floats. (Same problem as pixels in [[multimodal-llms]].)
2. It's absurdly long. A 30-second clip is ~480,000 numbers. A transformer's [[attention-and-transformers|attention]] is quadratic in sequence length, so feeding raw samples is a non-starter.

Every audio model is, at heart, a strategy for getting around those two problems. The field splits into three classic jobs, plus one new frontier:

• ASR — Automatic Speech Recognition (audio → text). "Transcribe what was said." The canonical modern system is [Whisper](https://arxiv.org/abs/2212.04356).
• TTS — Text-to-Speech (text → audio). "Read this sentence aloud in a natural voice." Modern TTS is neural and increasingly autoregressive — it predicts audio tokens the way an LLM predicts words.
• Audio codecs / tokenizers (audio ↔ discrete codes). The plumbing that makes the other two tractable: [SoundStream](https://arxiv.org/abs/2107.03312) and [EnCodec](https://arxiv.org/abs/2210.13438). This is the audio analogue of [[tokenization]].
• Speech LLMs / the audio-in-audio-out frontier (audio → audio, possibly while also reading and writing text). One model that listens to you and replies in spoken voice, with no separate ASR or TTS in the loop. [AudioLM](https://arxiv.org/abs/2209.03143) opened the door; [Moshi](https://kyutai.org/Moshi.pdf) and GPT-4o-style real-time voice walked through it.

A single mental anchor for the whole page: once sound is a sequence of discrete tokens, "speaking" is just next-token prediction. That is the load-bearing realization, and it is exactly the same realization that powers [[multimodal-llms]] for images.

Why it matters

• It's how voice assistants actually work — and how the good ones (the ones that interrupt naturally, don't talk over you, and answer in ~200 ms) differ from the bad ones.
• It unifies modalities. Audio tokenization (below) makes sound the same shape as text and image tokens. That's what lets one transformer be an any-to-any model — the dream behind [[multimodal-llms]] and behind Alice's own multimodal ambitions.
• The old pipeline is collapsing. For thirty years, voice systems were three boxes bolted together: ASR → text-LLM → TTS. Each box adds latency and throws away information (prosody, emotion, the fact that you started speaking). Speech LLMs fuse them, which is both a capability leap and an architectural earthquake.
• It's a clean lesson in tokenization-as-bottleneck. Just as text quality is gated by the [[tokenization|BPE tokenizer]], audio quality is gated by the codec. A model can only say what its codebook can encode.

How it works (real mechanics)

We'll build up in the natural dependency order: ASR (Whisper) → codecs/RVQ (SoundStream, EnCodec) → speech LLMs (AudioLM, Moshi). The codec is the keystone, so don't skip it.

1. ASR — Whisper, the encoder-decoder workhorse

Whisper↗ (Radford et al., 2022) is a vanilla encoder-decoder transformer ([[attention-and-transformers]]) — the same shape as the original "Attention Is All You Need" model. Its cleverness is not architectural; it's in the data and the task framing.

Input. Whisper does not eat raw samples. It first converts 30 seconds of 16 kHz audio into a log-mel spectrogram: an image-like 2D array of (time × frequency-band) energies, 80 mel bins, ~3000 time frames. A small convolutional stem then turns this into the sequence the encoder consumes. So Whisper's "tokenization of audio" is a fixed, classical signal-processing transform (the mel spectrogram), not a learned codec — an important contrast with everything below.

Training data — the actual innovation. 680,000 hours of audio scraped from the internet with their (weak, noisy, human-or-machine-generated) transcripts. No careful curation, no gold labels — large-scale weak supervision. The bet (which paid off): enough messy data beats a little clean data, and the model learns to be robust to real-world audio (accents, noise, domains) rather than overfitting one benchmark. The headline claim is strong zero-shot performance — competitive transcription on datasets it never trained on, without fine-tuning, approaching human robustness.

Multitask via special tokens. Whisper folds many jobs into one decoder by prepending control tokens to the output sequence — exactly the [[tool-use-function-calling|"everything is a sequence"]] trick. The decoder is prompted with a sequence like:

<|startoftranscript|> <|language:en|> <|transcribe|> <|notimestamps|>  ... text tokens ...

Swap <|transcribe|> for <|translate|> and the same model translates speech to English text. Include timestamp tokens and it emits word-/segment-level timings. Swap the language tag and it transcribes another language. One model, one decoder, many tasks — all selected by special tokens, never by changing weights.

Honest limitation: Whisper's 30-second window and its decoder make it prone to hallucinated transcripts on silence or non-speech audio, and to repetition loops. It is a transcriber, not an understander — it does not know meaning, only the mapping from sound to text.

2. The keystone — neural audio codecs and RVQ

To make audio behave like text for a transformer, we need a learned tokenizer that maps a waveform to a short sequence of integers (codes) and back, losing as little perceptual quality as possible. This is a neural audio codec, and the dominant design is Residual Vector Quantization (RVQ), introduced for this purpose by SoundStream↗ (Zeghidour et al., 2021) and refined by EnCodec↗ (Défossez et al., 2022).

The skeleton is a fully-convolutional encoder-decoder, trained end-to-end:

waveform ──Encoder(conv,strided)──▶ z  (continuous latent, ~75 vectors/sec)
   z ──Quantizer──▶ discrete codes  (integers — THIS is the audio "tokenization")
codes ──Decoder(conv,transposed)──▶ reconstructed waveform

The encoder downsamples aggressively (the strided convolutions reduce ~16,000 samples/sec to ~50–75 latent frames/sec), solving the "absurdly long" problem. The quantizer solves the "continuous" problem by snapping each latent vector to the nearest entry in a learned codebook — turning a float vector into an integer index, just like an embedding lookup run backwards.

Why *residual* VQ? A single codebook of feasible size (say 1024 entries = 10 bits) cannot capture all the nuance in a latent vector — the reconstruction would be crude. RVQ fixes this with a cascade of quantizers, each cleaning up the previous one's error. The math is simple and worth internalizing:

r₀ = z                                  # start with the full latent vector
for i = 1 .. Q:                         # Q quantizer stages (codebooks)
    cᵢ  = nearest_codebook_entry(Cᵢ, rᵢ₋₁)   # quantize the current residual
    rᵢ  = rᵢ₋₁ − cᵢ                      # what's left over becomes the next residual
ẑ = c₁ + c₂ + ... + c_Q                 # final reconstruction = sum of all stages

Stage 1 gives a coarse approximation; stage 2 quantizes what stage 1 missed; stage 3 quantizes what stages 1–2 missed; and so on. With Q codebooks of N entries each, you get the expressive power of roughly NᏃ effective codewords for the storage cost of only Q×N codebook entries — an exponential win. Each frame of audio is thus represented not by one token but by a stack of Q tokens (one per codebook level): the first is the most important ("coarse"), the last is fine detail. Remember this stack — it is exactly what makes speech LLMs hard.

Bitrate scalability via quantizer dropout. During training, SoundStream randomly drops the later quantizers for some examples ("structured dropout" / quantizer dropout). This forces the early codebooks to carry a usable signal on their own, so a single trained model can run at many bitrates: use all Q codebooks for high fidelity, or just the first few for a low-bitrate stream. EnCodec reaches roughly 1.5–24 kbps at 24 kHz this way.

Training losses combine a reconstruction term (multi-scale spectrogram L1/L2 — "does it sound right across frequency scales?") with a GAN adversarial loss (a discriminator that learns to tell real audio from reconstructed, pushing the decoder toward realism — see [[diffusion-models]] for the generative-modeling context). EnCodec adds a clever loss balancer: instead of hand-tuning loss weights, the weight of each loss defines the fraction of the total gradient it contributes, which stabilizes a notoriously finicky training recipe.

The big idea, stated plainly: a neural codec is to audio what [[tokenization|BPE]] is to text. It defines the vocabulary the rest of the stack is allowed to use. Everything downstream inherits its ceiling.

3. Two flavors of audio token: semantic vs. acoustic

There's a subtle problem the codec alone doesn't solve. RVQ tokens are optimized for faithful reconstruction — they encode how the audio sounds (timbre, speaker identity, room reverb). But they're a poor substrate for long-range language structure — you can't easily predict the next 10 seconds of coherent speech from raw acoustic codes; they drift.

AudioLM↗ (Borsos et al., 2022) made the foundational distinction:

• Semantic tokens — extracted from a self-supervised speech model (w2v-BERT). These capture linguistic and structural content: phonetics, syntax, long-term coherence. Good for "what is being said," bad for "exactly how it sounds."
• Acoustic tokens — the SoundStream RVQ codes. These capture fine detail: speaker voice, prosody, recording conditions. Good for high-fidelity synthesis, bad for long-range planning.

AudioLM then models audio hierarchically with a chain of transformers:

Stage 1: predict SEMANTIC tokens             → long-term structure / "the plan"
Stage 2: predict COARSE acoustic tokens | semantic   → fill in voice, roughly
Stage 3: predict FINE acoustic tokens   | coarse     → fill in fidelity details

Each stage is plain next-token prediction (an LLM, but over audio codes). The semantic stage keeps the output coherent over time; the acoustic stages make it sound real. This semantic/acoustic split is the conceptual bridge from "codec" to "speech LLM," and it recurs everywhere downstream.

4. The frontier — speech LLMs and audio-in / audio-out (Moshi)

Now the synthesis. A speech LLM is a transformer that consumes and produces audio tokens directly, so a spoken conversation never has to round-trip through text. Moshi↗ (Défossez et al., Kyutai, 2024) is the clearest open example and shows every moving part.

Components:

• Helium — a 7B-parameter text [[attention-and-transformers|transformer]] backbone (RoPE positional encoding — see [[positional-encoding]] — gated linear units, FlashAttention — see [[flash-attention]]), pretrained on ~2.1T text tokens. This is the "brain" / reasoning core.
• Mimi — Moshi's own streaming neural codec, running at a startlingly low 12.5 Hz frame rate (~1.1 kbps) with 8 codebooks of 2048 entries. Crucially, Mimi uses a "split RVQ": codebook #1 is distilled to match WavLM self-supervised features (making it a semantic token), while the remaining 7 levels are ordinary RVQ acoustic tokens. So Mimi delivers AudioLM's semantic/acoustic split inside one streaming codec.
• RQ-Transformer — the generator that handles the "stack of tokens per frame" problem (recall §2). It factorizes generation along two axes: - a large temporal transformer (32 layers) that steps across time frames (the slow axis), and - a small depth transformer (6 layers) that, within one frame, autoregressively emits the 8 codebook levels (the fast axis, semantic → acoustic). This two-level factorization is the key trick: it lets the model produce a whole stack of codes per timestep without making the main sequence 8× longer.

Full-duplex, multi-stream. A real conversation is not turn-by-turn; people overlap, interrupt, backchannel ("mm-hm"). Moshi models two audio streams jointly — the user's and Moshi's own — at every timestep. Because Moshi always models its own output stream, it can decide to start speaking, stop, or talk over you. There is no "now it's your turn" handoff. This is what "full-duplex" means and why it feels alive.

Inner Monologue — the part that makes it smart. Pure audio-to-audio generation tends to be linguistically weak (the acoustic objective doesn't pressure grammar). Moshi's fix: at each frame it predicts a time-aligned text token first, as a prefix to that frame's audio tokens. The model literally "thinks in text, a beat before it speaks." Empirically this is enormous — it drops generation negative-log-likelihood from 4.36 → 2.77 and lifts spoken-QA accuracy (e.g. WebQ from 9% → 26.6%). It's a beautiful demonstration that the [[test-time-compute-reasoning|reasoning lives in the text channel]] even when the output is voice.

Latency. Because everything is one streaming model with no ASR/LLM/TTS pipeline, Moshi hits a theoretical ~160 ms end-to-end latency (~200 ms in practice) — fast enough to feel like a real conversation. GPT-4o's voice mode (Realtime API, late 2024) targets the same native audio-in/audio-out regime.

A compact picture of the whole stack:

       YOU SPEAK                                MODEL SPEAKS
   waveform                                        waveform
      │ Mimi encoder                                  ▲ Mimi decoder
      ▼                                               │
  user audio tokens ──┐                  ┌── model audio tokens (8 codebooks/frame)
                      ▼                  │            ▲ depth transformer (within frame)
              ┌──────────────────────────────────┐
              │  RQ-Transformer (Helium backbone) │  ← also emits text "inner monologue"
              │  temporal transformer over frames │
              └──────────────────────────────────┘
                jointly models BOTH streams → full-duplex

Key ideas & tradeoffs

The unifying principle: audio modeling is a chain of lossy compressions — waveform → latent → discrete codes → (semantic vs acoustic) → tokens an LLM predicts. Each link trades fidelity for tractability. Quality is bounded by the weakest link, and for most modern systems that link is the codec's codebook.

Honest caveats

• The codec is the ceiling. A speech LLM can only utter sounds its codec can encode. Artifacts, metallic timbre, and limited speaker range usually trace to the codec, not the LLM. This is the audio twin of the [[tokenization|"the tokenizer is why it can't spell"]] problem.
• Whisper hallucinates. On silence, music, or non-speech, Whisper happily emits fluent-but-fabricated transcripts and repetition loops. It has no notion of "I didn't hear words." Never trust a transcript on out-of-distribution audio without a confidence/VAD gate.
• Native speech LLMs are hard to control and evaluate. A pipeline lets you log the intermediate text, swap the reasoning LLM, insert business logic, and red-team each stage. A native audio-to-audio model is opaque end-to-end — harder to debug, harder to align (see [[rlhf-and-alignment]]), and harder to benchmark (no clean text checkpoint). This is the main reason most 2026 production voice deployments still use pipelines despite the latency cost.
• Latency numbers are theoretical-best. "160 ms" assumes ideal streaming on capable hardware. Network, batching, and CPU-only inference (no GPU, as on OpenAlice's blal.de box) push real latency much higher.
• Data & licensing are murky. Whisper trained on 680k hours of scraped internet audio; codecs and TTS train on large speech corpora. Voice cloning raises real consent/impersonation concerns. "It works" ≠ "it's clean to ship."
• Tokenization isn't free of bias. Mel features and codecs are tuned on dominant languages/accents; ASR error rates and TTS naturalness degrade on under-represented speech. Robustness claims are averages, not guarantees for your users.
• Source honesty note: the arXiv pages returned only abstracts, and the Moshi PDF failed to parse as text (binary). The deep mechanics above (RVQ math, AudioLM's three-stage chain, Mimi's split-RVQ/8-codebooks/12.5 Hz, the RQ-Transformer's 32+6 layers, Inner Monologue's 4.36→2.77 NLL, ~160 ms latency) were cross-checked against secondary technical summaries and the AudioLM blog/PDF. Treat exact figures as "as reported," not independently re-derived here.

How it connects to OpenAlice + the Academy ladder

Where it sits in the ecosystem. Alice is meant to be a multimodal companion — she already reads (text), and the natural next axis is hearing and speaking. Everything on this page is the substrate for a future Alice voice loop. The relevant design choice for OpenAlice is the pipeline-vs-native tradeoff in the table above:

• A pipeline (Whisper-style ASR → Alice's existing text LLM via the provider config → neural TTS) is the pragmatic near-term path. It reuses Alice's whole text brain, personality envelope, memory, and safety gates unchanged — voice becomes just two adapters at the edges. Given the "preserve Alice's soul verbatim" and "files-first, no premature abstraction" discipline, this is the low-risk first step, and it fits the GPU-less blal.de reality (Whisper + a light TTS run on CPU; a native 7B+ speech LLM does not).
• A native speech LLM (Moshi/GPT-4o-style) is the frontier prize — full-duplex, sub-second, emotionally present — but it's an opaque model that's hard to align to Alice's character and hard to evaluate. That's a lab-branch experiment, not an mvp cutover.

The Academy ladder. This article is a capstone of the multimodal rung. Climb to it in order:

1. [[tokenization]] — text becomes integers. The single most important prerequisite: audio codecs are "tokenization for sound."
2. [[embeddings]] — integers become vectors a network can mix.
3. [[attention-and-transformers]] — the engine that predicts the next token (text or audio). Plus [[positional-encoding]] and [[flash-attention]] for the efficiency tricks Moshi's backbone uses.
4. [[multimodal-llms]] — the sibling of this page: pixels → tokens. Read it alongside; audio and vision are the same bridge built twice.
5. This page — sound → tokens → next-token prediction, all the way to a model that talks back.
6. Then branch outward: [[diffusion-models]] (an alternative generative route for audio synthesis), [[rlhf-and-alignment]] (how you'd align a voice model to Alice's character), and [[test-time-compute-reasoning]] (why Moshi's "inner monologue" text channel is where the thinking happens).

The throughline of the whole ladder, said once more: find a tokenization, and a transformer will model anything. Text proved it, images repeated it, audio confirms it.