研究音转唇项目

https://github.com/Rudrabha/Wav2Lip

1. 下载代码

https://github.com/Rudrabha/Wav2Lip

2. 安装相应的包

docker 用paddlepaddle的包

pip install -r requirements.txt

3. 在filelists文件夹下建main文件夹，里面放经过syncnet处理后的video数据，处理成mp4文件

4. 训练

训练音唇同步

python color_syncnet_train.py --data_root ./preprocessed/filelistsmain --checkpoint_dir ./checkpoints/new --checkpoint_path

loss要收敛到约0.25

训练生成器

python hq_wav2lip_train.py --data_root ./preprocessed/filelistsmain --checkpoint_dir ./checkpoints/new --syncnet_checkpoint_path ./checkpoints/lipsync_expert.pth --checkpoint_path ./checkpoints/wav2lip_gan.pth  --disc_checkpoint_path ./checkpoints/visual_quality_disc.pth

loss收敛到约0.2

4. 推断

python inference.py --checkpoint_path ./checkpoints/new/checkpoint_step000261000.pth --face ./work/video6.mp4  --audio ./work/video6.wav

GitHub - OpenTalker/SadTalker: [CVPR 2023] SadTalker：Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

Wav2Lip-GFPGAN/GFPGAN-master at main · ajay-sainy/Wav2Lip-GFPGAN · GitHub