---
description: >-
  Xie et al. "Mini-Omni: Language Models Can Hear, Talk While Thinking in
  Streaming". arXiv preprint arXiv:2408.16725 (2024).
---

# Reading Note: "Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming"

<figure><img src="https://725511345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fo36vbcqVTETuSOVJefc3%2Fuploads%2FNu8SmuBYoETnYlCbIsZj%2F000.png?alt=media&#x26;token=fcc1919c-857b-4aab-a45b-fd7d615ea1bc" alt="" width="563"><figcaption></figcaption></figure>

## Intro

_**Motivations**_:

* Human-computer interaction needs the model to <mark style="color:orange;">**directly reason with the audio modality**</mark> and generate output in streaming.
* Previous works typically depend on <mark style="color:orange;">**extra TTS systems**</mark> for speech synthesis, resulting in <mark style="color:orange;">**latency**</mark>.

_**Contributions**_:

* Mini-Omni: The <mark style="color:green;">**first audio-based end-to-end open-source**</mark> conversational model for <mark style="color:green;">**real-time**</mark> speech interaction.
* A <mark style="color:blue;">text-instructed speech generation</mark> method, along with <mark style="color:blue;">batch-parallel</mark> strategies during inference.
* A <mark style="color:blue;">VoiceAssistant-400K dataset</mark> generated by GPT-4o for speech SFT (most QA datasets contain <mark style="background-color:yellow;">mixed code or overly lengthy
  &#x20;text</mark>, rendering them <mark style="background-color:yellow;">unsuitable for speech</mark> models).

_**Challenges**_:

* _<mark style="color:blue;">Complexity of Audio Reasoning</mark>_: <mark style="color:orange;">**Direct training for audio modality**</mark> <mark style="color:orange;">**reasoning**</mark> often results in <mark style="color:orange;">**incoherent outputs**</mark> from the model.
* _<mark style="color:blue;">Model Complexity</mark>_: <mark style="color:orange;">**Incorporating additional modules**</mark> for speech input and output.
* _<mark style="color:blue;">Difficulty in Modality Alignment</mark>_: Reasoning ability developed for text is difficult to <mark style="color:orange;">**transfer**</mark> to audio.
* _<mark style="color:blue;">Resource Consumption</mark>_: Adapting model capability from text
  &#x20;to speech modality requires <mark style="color:orange;">**converting all data labels into audio**</mark> and retraining.

_**Characteristic**_:

* Adapting <mark style="background-color:yellow;">currently available methods</mark> for <mark style="color:blue;">discretizing speech tokens</mark> and employing the <mark style="background-color:yellow;">simplest model architecture</mark> for easier follow-up research.
* Using only a <mark style="color:orange;">**0.5B model**</mark> and a limited amount of synthesized audio data.
* Including various other audio-text functionalities such as <mark style="color:blue;">ASR</mark> and <mark style="color:blue;">TTS</mark>.

## "Any Model Can Talk": Three-Stage Training

_**Core Idea**_: The <mark style="color:orange;">**primary modality alignment**</mark> tasks are handled during <mark style="color:orange;">**adapter training**</mark>, thus the <mark style="color:green;">**original model’s capabilities are maximally preserved**</mark>.

<figure><img src="https://725511345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fo36vbcqVTETuSOVJefc3%2Fuploads%2FrjpNIeJsRDJNIrSz3shY%2F000.png?alt=media&#x26;token=2dd19a2f-e2f5-4b8c-8311-070db5cf3d9d" alt="" width="563"><figcaption></figcaption></figure>

**Stage 1: Modality Alignment**

* _**Goal**_: Enhancing the <mark style="background-color:yellow;">text model’s ability</mark> to understand (i.e., <mark style="color:orange;">**speech recognition**</mark>) and generate (i.e., <mark style="color:orange;">**speech synthesis**</mark>) speech, aligning the speech modality with the text model’s input.
* _**Data**_: From <mark style="color:blue;">speech recognition</mark> and <mark style="color:blue;">speech synthesis</mark> tasks.

**Stage 2: Adaption Training**

* _**Goal**_: Training the model’s <mark style="color:orange;">**text capabilities**</mark> when <mark style="color:orange;">**given audio inputs**</mark>, as <mark style="background-color:yellow;">audio output is simply synthesized from text</mark>.
* _**Data**_: From <mark style="color:blue;">speech recognition</mark>, <mark style="color:blue;">spoken question answering</mark>, and <mark style="color:blue;">text response</mark> tasks.

**Stage 3: Multi-Modal Fine-Tuning**

* _**Goal**_: Fine-tuning the <mark style="color:orange;">**entire model**</mark>.
* _**Data**_: <mark style="color:orange;">**Comprehensive**</mark> data.

(Additional Final Stage: <mark style="color:orange;">**Annealing**</mark> and fine-tuning with <mark style="color:blue;">Voice QA</mark>.)

## Batch Parallel Decoding

<figure><img src="https://725511345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fo36vbcqVTETuSOVJefc3%2Fuploads%2FjmxL7KarRrkXBrJcVl6t%2F000.png?alt=media&#x26;token=4e09029a-6446-45e4-ba1c-5f145a63b6bd" alt=""><figcaption></figcaption></figure>

## Model Input

* Given the eight parallel output sequences, the input <mark style="color:orange;">**also requires eight sequences**</mark>.
* Placing the special token <mark style="color:green;">**\<answer>**</mark> in different positions to <mark style="color:orange;">**guide the model for multi-modal output**</mark>.

<figure><img src="https://725511345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fo36vbcqVTETuSOVJefc3%2Fuploads%2Fjuw5oycyygeFJ2hj7Tjk%2F000.png?alt=media&#x26;token=14f35423-9edb-4704-be40-440d2040078f" alt=""><figcaption></figcaption></figure>

## Dataset

<figure><img src="https://725511345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fo36vbcqVTETuSOVJefc3%2Fuploads%2FQvh2o4PiyQR2bDtkNZvF%2F000.png?alt=media&#x26;token=426ab270-e59e-4625-808b-1598803c84a9" alt="" width="563"><figcaption></figcaption></figure>

