【DreamTalk】源码部署

最新推荐文章于 2024-05-27 17:00:50 发布

太空眼睛

最新推荐文章于 2024-05-27 17:00:50 发布

阅读量700

点赞数 5

文章标签： dlib dreamtalk 数字人 python ffmpeg opencv 文本生成视频

本文链接：https://blog.csdn.net/friendlytkyj/article/details/136482648

版权

安装

# 下载源码
git clone https://github.com/ali-vilab/dreamtalk
cd dreamtalk

conda create -n dreamtalk python=3.10
conda activate dreamtalk

conda install -c conda-forge yacs==0.1.8
conda install -c conda-forge numpy==1.21.5
conda install -c conda-forge av==10.0.0
conda install ffmpeg

# 修改requirements.txt中opencv-python的版本: opencv-python==4.9.0.80
pip install -r requirements.txt

# CPU版
# conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 cpuonly -c pytorch
# GPU版(https://pytorch.org/get-started/locally/)
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia

pip install urllib3==1.26.6
pip install transformers==4.28.1
conda install -c conda-forge dlib
pip install chardet
conda install -c conda-forge blas

模型下载

下载到`dreamtalk/checkpoints`目录

https://modelscope.cn/api/v1/models/iic/dreamtalk/repo?Revision=master&FilePath=checkpoints/denoising_network.pth
https://modelscope.cn/api/v1/models/iic/dreamtalk/repo?Revision=master&FilePath=checkpoints/renderer.pt

下载到`dreamtalk/AI-ModelScope`

cd dreamtalk/AI-ModelScope
git clone https://www.modelscope.cn/AI-ModelScope/wav2vec2-large-xlsr-53-english.git

修改inference_for_demo_video.py

WAV2VEC_MODEL_PATH = "/xxxx/dreamtalk/AI-ModelScope/wav2vec2-large-xlsr-53-english"

...

# get wav2vec feat from audio
wav2vec_processor = Wav2Vec2Processor.from_pretrained(WAV2VEC_MODEL_PATH)

wav2vec_model = (
    Wav2Vec2Model.from_pretrained(WAV2VEC_MODEL_PATH)
    .eval()
    .to(device)
)

测试验证

GPU运行

python inference_for_demo_video.py \
--wav_path /vxiao/funasr-runtime-resources/models/output.wav \
--style_clip_path data/style_clip/3DMM/M030_front_neutral_level1_001.mat \
--pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \
--image_path /vxiao/SadTalker/examples/source_image/art_5.png \
--cfg_scale 1.0 \
--max_gen_len 30 \
--output_name test01

运行结果

在这里插入图片描述

耗时29秒

CPU运行

python inference_for_demo_video.py \
--wav_path /vxiao/funasr-runtime-resources/models/output.wav \
--style_clip_path data/style_clip/3DMM/M030_front_neutral_level1_001.mat \
--pose_path data/pose/RichardShelby_front_neutral_level1_001.mat \
--image_path /vxiao/SadTalker/examples/source_image/art_5.png \
--cfg_scale 1.0 \
--max_gen_len 30 \
--device cpu \
--output_name test01