使用“阿里云人工智能平台 PAI”制作数字人

体验 阿里云人工智能平台 PAI

PAI-DSW免费试用

https://free.aliyun.com/?spm=5176.14066474.J_5834642020.5.7b34754cmRbYhg&productCode=learn

https://help.aliyun.com/document_detail/2261126.html

体验PAI-DSW

https://help.aliyun.com/document_detail/2261126.html

基于Wav2Lip+TPS-Motion-Model+CodeFormer技术实现动漫风数字人

只需要输入一张动漫形象图片，以及你想让它说的文本内容，它就可以准确的说出你的文本内容并模仿人类的说话动作！

动漫形象的生成可以参考开源代码库 EasyPhoto 、案例easyphoto_diffusers、stable_diffusion_easyphoto。

语音生成相关案例可以参考：ai_singer_rvc、ai_singer_svc。

环境准备

克隆开源代码（网络不稳克隆容易失败，建议多尝试几次。）

#注意：Wav2Lip开源库不可商用，仅用于教学。请遵纪守法，不要用于非法活动。
!git clone https://github.com/Rudrabha/Wav2Lip.git

!git clone https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model.git

!git clone https://github.com/sczhou/CodeFormer.git

注释掉codeformer中的import bug：

!sed -i 's/from .version/# from .version/' CodeFormer/basicsr/__init__.py

初始化工作目录：

import os.path as osp

WORKDIR = osp.abspath('.')
print(f'work directory: {WORKDIR}')
WAV2LIP_WORKDIR = osp.join(WORKDIR, 'Wav2Lip')
print(f'wav2lip directory: {WAV2LIP_WORKDIR}')
CODEFORMER_WORKDIR = osp.join(WORKDIR, 'CodeFormer')
print(f'codeformer directory: {CODEFORMER_WORKDIR}')
MOTION_MODEL_WORKDIR = osp.join(WORKDIR, 'Thin-Plate-Spline-Motion-Model')
print(f'motion model directory: {MOTION_MODEL_WORKDIR}')

安装依赖包

!pip install --upgrade pip && \
 pip install -r {MOTION_MODEL_WORKDIR}/requirements.txt --extra-index-url https://download.pytorch.org/whl/cu113/ && \
 pip install -r {CODEFORMER_WORKDIR}/requirements.txt && \
 pip install modelscope==1.10.0 pytorch_wavelets tensorboardX && \
 pip install kantts -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html && \
 pip install librosa==0.8.0 resampy --no-deps && \
 pip install sentencepiece && pip install --upgrade transformers && \
 sudo apt update -y && sudo apt install -y ffmpeg

如果遇到报错cannot import name ‘kaiser’ from ‘scipy.signal’，可以尝试对scipy进行降级，我在Terminal执行下述命令之后就可以继续执行依赖包的安装了

pip install --upgrade scipy==1.7.3

下载运行脚本

!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/codes/run.py -O run.py
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/codes/utils.py -O utils.py

预训练模型准备

下载wav2lip依赖模型并保存到指定目录。

!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/wav2lip/wav2lip.pth -O {WAV2LIP_WORKDIR}/checkpoints/wav2lip.pth
!mkdir -p ~/.cache/torch/hub/checkpoints/
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/wav2lip/s3fd-619a316812.pth -O ~/.cache/torch/hub/checkpoints/s3fd-619a316812.pth

下载Thin-Plate-Spline-Motion-Model依赖模型并保存到指定目录。

!mkdir -p {MOTION_MODEL_WORKDIR}/checkpoints
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/Thin-Plate-Spline-Motion-Model/vox.pth.tar -O {MOTION_MODEL_WORKDIR}/checkpoints/vox.pth.tar

下载codeformer依赖模型并保存到指定目录。

!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/codeformer/CodeFormer/codeformer.pth -O {CODEFORMER_WORKDIR}/weights/CodeFormer/codeformer.pth
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/codeformer/facelib/detection_Resnet50_Final.pth -O {CODEFORMER_WORKDIR}/weights/facelib/detection_Resnet50_Final.pth
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/codeformer/facelib/parsing_parsenet.pth -O {CODEFORMER_WORKDIR}/weights/facelib/parsing_parsenet.pth
!mkdir -p {CODEFORMER_WORKDIR}/weights/realesrgan
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/pretrained_models/codeformer/realesrgan/RealESRGAN_x2plus.pth -O {CODEFORMER_WORKDIR}/weights/realesrgan/RealESRGAN_x2plus.pth

运行

下载测试图片和驱动视频：

!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/imgs/anime_portrait.png -O anime_portrait.png
!wget -nc http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/projects/liveportrait/videos/drive_video.mp4 -O drive_video.mp4

查看测试图片和驱动视频：

from IPython.display import Image, display, Video

image_path = './anime_portrait.png'
raw_photo = Image(image_path)
display(raw_photo)

video_path = 'drive_video.mp4'
Video(video_path)

生成视频：

可通过–text参数指定文本内容自动生成语音，语音生成模型可参考魔搭社区中的相关语音模型修改–tts_model参数；或者通过–audio参数直接指定一段语音。

因为本案例中动作驱动使用了开源代码库Thin-Plate-Spline-Motion-Model，并默认使用relative模式生成动作，建议人像图片和驱动视频的首帧动作尤其是嘴部动作尽可能相似，才可能达到理想的效果。

!python run.py --raw_photo anime_portrait.png --drive_video drive_video.mp4 --out outputs --text "I think It is possible for ordinary people to choose to be extraordinary"

结果可视化：

from IPython.display import Image, display, Video

video_path = 'outputs/final_result.mp4'
Video(video_path)

结果视频
链接：https://pan.baidu.com/s/1eo_udVUg2M6SR-khbrWxKA?pwd=z76z
提取码：z76z