在实时语音交互上超过GPT-4o，端到端语音模型Mini-Omni部署

Mini-Omni是清华大学开源的多模态大型语言模型，具备实时语音输入和流式音频输出的能力。

Mini-Omni模型能够一边听、一边说，一边思考，类似于ChatGPT的语言对话模式。

Mini-Omni模型的主要特点是能够直接通过音频模态进行推理，并生成流式输出，而不需要依赖额外的文本到语音(TTS)系统，从而减少了延迟。

Mini-Omni模型的架构在Qwen2-0.5B基础上进行了增强，使用了Whisper-small编码器来有效处理语音输入。

Mini-Omni模型采用了并行文本-音频生成方法，通过批量并行解码生成语音和文本，确保了模型在不同模态间的推理能力不受损害。

Mini-Omni模型还引入了VoiceAssistant-400K数据集，用于对优化语音输出的模型进行微调。

github项目地址：https://github.com/gpt-omni/mini-omni。

一、环境安装

1、python环境

建议安装python版本在3.10以上。

2、pip库安装

pip install torch==2.3.1+cu118 torchvision==0.18.1+cu118 torchaudio==2.3.1 --extra-index-url https://download.pytorch.org/whl/cu118

pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

3、模型下载：

git lfs install

git clone https://huggingface.co/gpt-omni/mini-omni

二、功能测试

1、运行测试：

（1）python代码调用测试

import os
import torch
import time
import lightning as L
import soundfile as sf
import whisper
from snac import SNAC
from litgpt import Tokenizer
from tqdm import tqdm
from huggingface_hub import snapshot_download
from lightning.fabric.utilities.load import _lazy_load as lazy_load
from utils.snac_utils import layershift, reconscruct_snac, reconstruct_tensors, get_time_str, get_snac, generate_audio_data
from litgpt.utils import num_parameters
from litgpt.generate.base import generate_AA, generate_ASR, generate_TA, generate_TT, generate_AT, generate_TA_BATCH, next_token_batch
from litgpt.model import GPT, Config


torch.set_printoptions(sci_mode=False)


# Constants Definitions
text_vocabsize = 151936
text_specialtokens = 64
audio_vocabsize = 4096
audio_specialtokens = 64

padded_text_vocabsize = text_vocabsize + text_specialtokens
padded_audio_vocabsize = audio_vocabsize + audio_specialtokens

_eot = text_vocabsize
_pad_t = text_vocabsize + 1
_input_t = text_vocabsize + 2
_answer_t = text_vocabsize + 3
_asr = text_vocabsize + 4

_eoa = audio_vocabsize
_pad_a = audio_vocabsize + 1
_input_a = audio_vocabsize + 2
_answer_a = audio_vocabsize + 3
_split = audio_vocabsize + 4


# Utility Functions
def get_input_ids_TA(text, text_tokenizer):
    input_ids_item = [[] for _ in range(8)]
    text_tokens = text_tokenizer.encode(text)
    for i in range(7):
        input_ids_item[i] = [layershift(_pad_a, i)] * (len(text_tokens) + 2) + [
            layershift(_answer_a, i)
        ]
        input_ids_item[i] = torch.tensor(input_ids_item[i]).unsqueeze(0)
    input_ids_item[-1] = [_input_t] + text_tokens.tolist() + [_eot] + [_answer_t]
    input_ids_item[-1] = torch.tensor(input_ids_item[-1]).unsqueeze(0)
    return input_ids_item


def get_input_ids_TT(text, text_tokenizer):
    input_ids_item = [[] for i in range(8)]
    text_tokens = text_tokenizer.encode(text).tolist()
    for i in range(7):
        input_ids_item[i] = torch.tensor(
            [layershift(_pad_a, i)] * (len(text_tokens) + 3)
        ).unsqueeze(0)
    input_ids_item[-1] = [_input_t] + text_tokens + [_eot] + [_answer_t]
    input_ids_item[-1] = torch.tensor(input_ids_item[-1]).unsqueeze(0)
    return input_ids_item


def get_input_ids_whisper(mel, leng, whispermodel, device, 
                          special_token_a=_answer_a, special_token_t=_answer_t):
    with torch.no_grad():
        mel = mel.unsqueeze(0).to(device)
        audio_feature = whispermodel.embed_audio(mel)[0][:leng]

    T = audio_feature.size(0)
    input_ids = []
    for i in range(7):
        input_ids_item = []
        input_ids_item.append(layershift(_input_a, i))
        input_ids_item += [layershift(_pad_a, i)] * T
        input_ids_item += [(layershift(_eoa, i)), layershift(special_token_a, i)]
        input_ids.append(torch.tensor(input_ids_item).unsqueeze(0))
    input_id_T = torch.tensor([_input_t] + [_pad_t] * T + [_eot, special_token_t])
    input_ids.append(input_id_T.unsqueeze(0))
    return audio_feature.unsqueeze(0), input_ids


def get_input_ids_whisper_ATBatch(mel, leng, whispermodel, device):
    with torch.no_grad():
        mel = mel.unsqueeze(0).to(device)
        audio_feature = whispermodel.embed_audio(mel)[0][:leng]
    
    T = audio_feature.size(0)

    input_ids_AA, input_ids_AT = [], []
    for i in range(7):
        lang_shift = layershift(_pad_a, i)
        input_ids_item_AA = [layershift(_input_a, i)] + [lang_shift] * T + [(layershift(_eoa, i)), layershift(_answer_a, i)]
        input_ids_item_AT = [layershift(_input_a, i)] + [lang_shift] * T + [(layershift(_eoa, i)), lang_shift]
        input_ids_AA.append(torch.tensor(input_ids_item_AA))
        input_ids_AT.append(torch.tensor(input_ids_item_AT))

    input_id_T = torch.tensor([_input_t] + [_pad_t] * T + [_eot, _answer_t])
    input_ids_AA.append(input_id_T)
    input_ids_AT.append(input_id_T)