Facebookresearch-Slowfast配置及demo实现

写在前面：

本博客所有环境基于服务器Ubuntu18.05系统，V100显卡

一、环境准备

1、Python

conda create -n your-env-name python=3.8 -y

conda activate your-env-name

2、Pytorch

Slowfast要求torch版本高于1.7，本博客使用torch-1.12.1+cu113的配置，本文不使用pip命令进行下载，使用本地编译的方式，请在下方链接找到所需版本的torch，torchvision，torchaudio，下载完成后传输到服务器上进行安装

https://download.pytorch.org/whl/torch_stable.htmlhttps://download.pytorch.org/whl/torch_stable.html

 3、工具包安装

可以根据slowfast中INSTALL.md里的指示进行安装

3.1 numpy

pip install numpy

3.2 PyAV

conda install av -c conda-forge

3.3 ffmpeg

pip install ffmpeg

3.4 moviepy

pip install moviepy

 3.5 psutil

pip install psutil

3.6 opencv-python

pip install opencv-python

3.7  tensorboard

pip install tensorboard

3.8  simplejson

pip install simplejson

4、其余部分安装

4.1 fvcore

在Github上下载下来之后上传到服务器进行编译

GitHub - facebookresearch/fvcore: Collection of common code that's shared among different research projects in FAIR computer vision team.Collection of common code that's shared among different research projects in FAIR computer vision team. - facebookresearch/fvcorehttps://github.com/facebookresearch/fvcore下方命令请cd到fvcore所在目录之后运行

pip install -e .

fvcore、PyYaml、tqdm、iopath会一起安装，如果没有一起安装，请再次使用pip install命令

4.2 PytorchVideo

与4.1的fvcore一样，在Github上下载源码后进行编译安装

GitHub - facebookresearch/pytorchvideo: A deep learning library for video understanding research.A deep learning library for video understanding research. - facebookresearch/pytorchvideohttps://github.com/facebookresearch/pytorchvideo

pip install -e .

4.3 Detectron2

这个是本篇文章中最难安装的部分，请确保安装了G++以及gcc编译器

GitHub - facebookresearch/detectron2: Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.Detectron2 is a platform for object detection, segmentation and other visual recognition tasks. - facebookresearch/detectron2https://github.com/facebookresearch/detectron2

pip install -e .

这一部分会出现比较多的问题，我不太能记清我出现的问题，如果有问题请把错误代码发在评论区我一一解答

5、Slowfast安装

GitHub - facebookresearch/SlowFast: PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models. - facebookresearch/SlowFasthttps://github.com/facebookresearch/SlowFast注意：下载完成之后请打开setup.py，将PIL修改为pillow以及sklearn修改为scikit-learn

pip install scikit-learn
pip install pandas

 完成安装上面两个包之后，运行如下命令

python setup.py build develop

如果出现finish slowfast==1.0即为安装成功

二、测试

1、下载预训练权重

SLOWFAST_32x2_R101_50_50.pkl下载SLOWFAST_32x2_R101_50_50.pkl放到slowfast/demo/AVA目录下

2、建立行为json

在slowfast/demo/AVA目录下面创建一个叫做ava.json的文件，写入如下内容

{
  "bend/bow (at the waist)": 0,
  "crawl": 1,
  "crouch/kneel": 2,
  "dance": 3,
  "fall down": 4,
  "get up": 5,
  "jump/leap": 6,
  "lie/sleep": 7,
  "martial art": 8,
  "run/jog": 9,
  "sit": 10,
  "stand": 11,
  "swim": 12,
  "walk": 13,
  "answer phone": 14,
  "brush teeth": 15,
  "carry/hold (an object)": 16,
  "catch (an object)": 17,
  "chop": 18,
  "climb (e.g., a mountain)": 19,
  "clink glass": 20,
  "close (e.g., a door, a box)": 21,
  "cook": 22,
  "cut": 23,
  "dig": 24,
  "dress/put on clothing": 25,
  "drink": 26,
  "drive (e.g., a car, a truck)": 27,
  "eat": 28,
  "enter": 29,
  "exit": 30,
  "extract": 31,
  "fishing": 32,
  "hit (an object)": 33,
  "kick (an object)": 34,
  "lift/pick up": 35,
  "listen (e.g., to music)": 36,
  "open (e.g., a window, a car door)": 37,
  "paint": 38,
  "play board game": 39,
  "play musical instrument": 40,
  "play with pets": 41,
  "point to (an object)": 42,
  "press": 43,
  "pull (an object)": 44,
  "push (an object)": 45,
  "put down": 46,
  "read": 47,
  "ride (e.g., a bike, a car, a horse)": 48,
  "row boat": 49,
  "sail boat": 50,
  "shoot": 51,
  "shovel": 52,
  "smoke": 53,
  "stir": 54,
  "take a photo": 55,
  "text on/look at a cellphone": 56,
  "throw": 57,
  "touch (an object)": 58,
  "turn (e.g., a screwdriver)": 59,
  "watch (e.g., TV)": 60,
  "work on a computer": 61,
  "write": 62,
  "fight/hit (a person)": 63,
  "give/serve (an object) to (a person)": 64,
  "grab (a person)": 65,
  "hand clap": 66,
  "hand shake": 67,
  "hand wave": 68,
  "hug (a person)": 69,
  "kick (a person)": 70,
  "kiss (a person)": 71,
  "lift (a person)": 72,
  "listen to (a person)": 73,
  "play with kids": 74,
  "push (another person)": 75,
  "sing to (e.g., self, a person, a group)": 76,
  "take (an object) from (a person)": 77,
  "talk to (e.g., self, a person, a group)": 78,
  "watch (a person)": 79
}

按照下图修改slowfast/demo/AVA/SLOWFAST_32x2_R101_50_50.yaml文件内容

3、修改预训练权重路径

4、修改json文件路径

5、修改推理视频路径

其中input为待推理的视频路径，output为推理完成后视频存放的路径

6、推理命令

python tools/run_net.py --cfg /path/to/your/cfg/file

三、推理效果图