TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting 论文部署测试

Talking Gaussian

Ubuntu下CUDA11.3配置

参考

https://blog.csdn.net/weixin_71109434/article/details/130566689

sudo sh cuda_11.3.1_465.19.01_linux.run --override

安装GCC-9，G+±9

sudo apt install gcc-9 g++-9

sudo rm /usr/bin/gcc
sudo ln -s /usr/bin/gcc-9 /usr/bin/gcc

Installation

git clone git@github.com:Fictionarry/TalkingGaussian.git --recursive

conda env create --file environment.yml
conda activate talking_gaussian
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
pip install tensorflow-gpu==2.8.0

安装 tensorflow-gpu时，需要用下面的方式：

pip install tensorflow-gpu==2.8.0 -i https://mirrors.aliyun.com/pypi/simple/

Preparation

• Prepare face-parsing model and the 3DMM model for head pose estimation.

  bash scripts/prepare.sh
  

• Download 3DMM model from Basel Face Model 2009:

  # 1. copy 01_MorphableModel.mat to data_util/face_tracking/3DMM/
  # 2. run following
  cd data_utils/face_tracking
  python convert_BFM.py
  

• Prepare the environment for EasyPortrait:

  # prepare mmcv
  conda activate talking_gaussian
  pip install -U openmim
  mim install mmcv-full==1.7.1
  
  # download model weight
  cd data_utils/easyportrait
  wget "https://n-ws-620xz-pd11.s3pd11.sbercloud.ru/b-ws-620xz-pd11-jux/easyportrait/experiments/models/fpn-fp-512.pth"
  

在Preparation这个步骤中，在下面的链接下载Basel Face Model 2009，而是在下文链接：
Download 3DMM model from Basel Face Model 2009:

https://huggingface.co/wsj1995/sadTalker/tree/main

Okay，到这里我的环境和数据准备完了，开始下一步操作。

Video Dataset

Here

Pre-processing Training Video

• Run script to process the video.

  python data_utils/process.py data/<ID>/<ID>.mp4
  

开始执行视频处理， 又得下载文件，代码里面下载，又是卡死，只能手动下载了。

Downloading /home/ads/.tensorflow/models/deepspeech-0_1_0-b90017e8.pb.zip from https://github.com/osmr/deepspeech_features/releases/download/v0.0.1/deepspeech-0_1_0-b90017e8.pb.zip

下载完，要手动解压，放在/home/ads/.tensorflow/models/deepspeech-0_1_0-b90017e8.pb。

Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /home/ads/.cache/torch/hub/checkpoints/resnet18-5c106cde.pth

• Obtain Action Units

  Run FeatureExtraction in OpenFace, rename and move the output CSV file to data/<ID>/au.csv.

• Generate tooth masks

  export PYTHONPATH=./data_utils/easyportrait 
  python ./data_utils/easyportrait/create_teeth_mask.py ./data/<ID>
  

  https://jqjhya6sjf.feishu.cn/docx/Wb4odtbajovJJjxeKCjcOPemnJb