This post introduces Llama-Mimi, a speech language model developed through the activities of the Dialogue WG at LLM-jp.
Speech Language Models
Large language models (LLMs) have advanced rapidly in recent years. An LLM treats text as a sequence of tokens and learns a probability distribution over token sequences from a large corpus through next-token prediction. At inference time, it uses that distribution to generate text.
This approach applies not only to text but to any data that can be tokenized. A speech language model represents speech as a token sequence using a speech tokenizer, and is trained on that sequence with next-token prediction, just like an LLM. Common ways to tokenize speech include discretizing intermediate-layer features of a model trained with self-supervised learning (SSL) (e.g. HuBERT, WavLM) with k-means clustering, and using VQ-VAE, a variant of the VAE with a discrete latent representation. Since Lakhotia et al. proposed the speech language model GSLM, research on speech language models has been very active.
Llama-Mimi
Llama-Mimi is a speech language model that uses Llama 3 as its Transformer decoder and Mimi as its speech tokenizer. While the architectures of speech language models have been growing more complex, LLMs and VLMs achieve high performance with simple architectures. With that in mind, we aimed to build a speech language model with an architecture that is as simple as possible.
Mimi is a kind of Unified Speech Tokenizer that tokenizes speech frame by frame. Each frame is represented by multiple tokens rather than a single one: the first token carries mostly phonetic information, and the subsequent tokens carry mostly acoustic information.
Llama-Mimi serializes these tokens and feeds them into the Transformer decoder. To feed speech tokens into Llama, we add them to Llama’s vocabulary. This also makes it possible to train on data that mixes text and speech in the future.
Training
We trained Llama 3 1B and 8B models on 240k hours of English speech data.
Once speech is tokenized with Mimi, the model can be trained just like an LLM, so we built our codebase by modifying TorchTitan, an LLM training framework, and trained the models with it.
The code is available on GitHub, so anyone can train Llama-Mimi.
Evaluation
In our paper, we evaluated Llama-Mimi with both likelihood-based and generation-based methods. Llama-Mimi achieved state-of-the-art performance on the acoustic consistency task, which asks a model to distinguish normal speech from anomalous speech in which, for example, the emotion or the speaker’s gender changes partway through. On the other hand, on semantic tasks, which require handling more global information, it performed slightly worse than models that use only semantic tokens, such as TWIST. We attribute this to the longer token sequences, which make long-range dependencies harder to capture. See the paper for details.
Generation Examples
Given a speech prompt, the trained Llama-Mimi can generate a continuation of it.
Below are examples generated by Llama-Mimi 1B and 8B when given the spoken prompt “It was a great day. It started.”
| Prompt | Llama-Mimi 1B | Llama-Mimi 8B |
|---|---|---|
The larger model generates more natural speech. More generation examples and comparisons with other models are available on our demo site.
Our model is available on Hugging Face. Give it a try!
Conclusion
This post introduced Llama-Mimi, a speech language model developed at LLM-jp. Compared with text language models, speech language models still struggle to generate long speech that stays semantically coherent. But considering how far models came from GPT-2 to GPT-4, this is a field whose future development is very exciting.