关于一些坑的故事

​

问题描述：

VS-Code的python插件问题:

The Python Tools server crashed 5 times in the last 3 minutes. The server will not be restarted.

解决方法：

https://github.com/microsoft/vscode-python/issues/3977

Turning on the language server

In your settings.json, set "python.languageServer": "Microsoft". You will then be prompted to restart VS Code. If you have not downloaded the language server before then that will be triggered when you trigger the extension by opening a Python file.

Issues directly related to the language server -- e.g. completions -- should be reported to the Microsoft/python-language-server issue tracker.

翻译过来就是在 .vscode/settings.json里，加一句 "python.languageServer": "Microsoft"，就会安装python相关的文件分析服务器。

Docker报错问题：

ImportError: libGL.so.1: cannot open shared object file: No such file or directory——docker

解决方法：

环境是ubuntu16.04。

指令：

sudo apt update
sudo apt install libgl1-mesa-glx

Anaconda下载包速度慢：

https://repo.anaconda.com/archive/

解决方法：

境内下载速度慢，修改为 清华源 中科大源：

清华源速度慢，而且之前闹过授权问题，所以现在换成了中科大源

直接打开cmd输入一下命令

# conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.ustc.edu.cn/anaconda/pkgs/free/
conda config --set show_channel_urls yes

包括下面的地址

# https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
# https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
# https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/
# https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/fastai/
# https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
# https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/

或者直接修改~/.condarc：

sudo vim ~/.condarc

改成如下

channels:
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/free/
show_channel_urls: true
ssl_verify: false

注意，如果使用了清华源，要删除，执行：

conda config --remove channels
https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/

conda install: Segmentation fault(段错误)：

之前用清华源，安装的时候网络不良导致中途出错，于是有些包虽然在本地缓存了，但其实不完整。
解决方法：删除所有缓存：

conda clean -a

其他的源：

# 创建名为 .condarc 的文件
conda config --set show_channel_urls yes 
# 清除索引缓存，保证用的是镜像站提供的索引
conda c lean -i
# 豆瓣源
conda config --add channels https://pypi.douban.com/anaconda/cloud/conda-forge/
conda config --add channels https://pypi.douban.com/anaconda/cloud/msys2/
conda config --add channels https://pypi.douban.com/anaconda/cloud/bioconda/
conda config --add channels https://pypi.douban.com/anaconda/cloud/menpo/
conda config --add channels https://pypi.douban.com/anaconda/cloud/pytorch/
# 清华源
channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud


channels:
  - defaults
custom_channels:
  conda-forge: https://mirrors.bfsu.edu.cn/anaconda/cloud
  msys2: https://mirrors.bfsu.edu.cn/anaconda/cloud
  bioconda: https://mirrors.bfsu.edu.cn/anaconda/cloud
  menpo: https://mirrors.bfsu.edu.cn/anaconda/cloud
  pytorch: https://mirrors.bfsu.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.bfsu.edu.cn/anaconda/cloud
default_channels:
  - https://mirrors.bfsu.edu.cn/anaconda/pkgs/main
  - https://mirrors.bfsu.edu.cn/anaconda/pkgs/r
  - https://mirrors.bfsu.edu.cn/anaconda/pkgs/msys2
show_channel_urls: True

-i https://pypi.doubanio.com/simple/  --trusted-host pypi.doubanio.com

channels:
  - defaults
show_channel_urls: true
channel_alias: https://anaconda.mirrors.sjtug.sjtu.edu.cn/
default_channels:
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/main
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/free
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/mro
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/msys2
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/pro
  - https://anaconda.mirrors.sjtug.sjtu.edu.cn/pkgs/r
custom_channels:
  conda-forge: https://anaconda.mirrors.sjtug.sjtu.edu.cn/conda-forge
  soumith: https://anaconda.mirrors.sjtug.sjtu.edu.cn/cloud/soumith
  bioconda: https://anaconda.mirrors.sjtug.sjtu.edu.cn/cloud/bioconda
  menpo: https://anaconda.mirrors.sjtug.sjtu.edu.cn/cloud/menpo
  viscid-hub: https://anaconda.mirrors.sjtug.sjtu.edu.cn/cloud/viscid-hub
  atztogo: https://anaconda.mirrors.sjtug.sjtu.edu.cn/cloud/atztogo

还有阿里源

TensorFlow的Keras路径问题：

TensorFlow的1.13，1.14，1.15对应的Keras位置各不相同，有的在tensorflow_core里，有的在tensorflow.python里，而tensorflow.python的路径也不一定相同。容易go to definition不了。

compile roi_align roi_pool 时候，DeprecatedTypeProperties & type() const { ^~~~ error: command ‘gcc’ failed with exit status 1

https://github.com/pytorch/pytorch/pull/20426
https://github.com/open-mmlab/mmdetection/issues/4610

解决办法：

Replace AT_CHECK with TORCH_CHECK [shard 1/10]
Rename AT_ASSERT to TORCH_INTERNAL_ASSERT; other macro updates
在roi_align_cuda.cpp里找到 AT_CHECK 改成 TORCH_CHECK

apex

错误一：

pip install apex

pip的apex和nvidia的apex没有任何关系，只是名字相同。

git clone https://github.com/NVIDIA/apex 
cd apex 
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

错误二：
pytorch和cuda版本没有对齐

Cleaning up...
  Removing source in /tmp/pip-req-build-v0deounv
Removed build tracker '/tmp/pip-req-tracker-3n3fyj4o'
ERROR: Command errored out with exit status 1: /users4/zsun/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-v0deounv/setup.py'"'"'; __file__='"'"'/tmp/p
ip-req-build-v0deounv/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_e
xt --cuda_ext install --record /tmp/pip-record-rce1cb4d/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
Exception information:

nvidia-smi的bug，没有process，gpu显存被占用问题：

ps -elf | grep python

BFM model 创建BFM.mat的问题：

https://github.com/YadiraF/face3d/tree/master/examples/Data/BFM
首先数据

# download those codes and data
# ...

# copy them into appropriate location
cd face3d/examples/Data/BFM
mkdir 3ddfa
cd 3ddfa
cp ./3DDFA_Release/Matlab/ModelGeneration/model_info.mat .
cp ./3DDFA_Release/Matlab/Model_Expression.mat .
cp ./FaceProfilingRelease_v1.1/FaceProfilingRelease/Model_tri_mouth.mat .
cp ./FaceProfilingRelease_v1.1/FaceProfilingRelease/Model_face_contour_trimed.mat .
cp ./CVPR2015_HPEN/High-Fidelity_PEN/Modelplus_nose_hole.mat .
cp ./CVPR2015_HPEN/High-Fidelity_PEN/Modelplus_parallel.mat .
cp ./3DDFA_Release/Matlab/vertex_code.mat .

然后是不用matlab，用octvae docker去build代码。

https://github.com/LucienXian/octave_run_BFM_docker
注意1需要到runBFM目录下跑docker run blabla
注意2需要修改"/Out/pncc_code.mat"成"/source/Out/pncc_code.mat"

dlib的GPU支持

git clone https://github.com/davisking/dlib.git
cd dlib
mkdir build
cd build
cmake .. -DDLIB_USE_CUDA=1 -DUSE_AVX_INSTRUCTIONS=1
cmake --build .
cd ..
python setup.py install --set USE_AVX_INSTRUCTIONS=1 --set DLIB_USE_CUDA=1

linux服务器查看进程数量

ps -ef | wc -l
ps -eLf | wc -l

tensorflow cuda 10.2的不兼容问题

10.2兼容10.1的内容，只需要把超链接给过去

sudo ln -s /usr/local/cuda/lib64/libcudart.so.10.2 /usr/local/cuda/lib64/libcudart.so.10.1
sudo ln -s /usr/local/cuda/extras/CUPTI/lib64/libcupti.so.10.2 /usr/local/cuda/extras/CUPTI/lib64/libcupti.so.10.1
#或者
sudo ln -s /usr/local/cuda/lib64/libcupti.so.10.2 /usr/local/cuda/lib64/libcupti.so.10.1

# cudnn 对应cuda 10.2的版本
$cd cuda
將檔案複製到 /usr/local/cuda
$sudo cp lib64/lib* /usr/local/cuda/lib64/
$sudo cp include/cudnn.h /usr/local/cuda/include/
$cd /usr/local/cuda/lib64/
#刪除原本鏈結
$sudo rm -rf libcudnn.so
$sudo rm -rf libcudnn.so.7
#新增新鏈結
$sudo chmod +r libcudnn.so.7.6.5
$sudo ln -s libcudnn.so.7.6.5 libcudnn.so.7
$sudo ln -s libcudnn.so.7 libcudnn.so

cannot import name ‘container_abcs’ from ‘torch._six’

import collections.abc as container_abcs

ERROR: Cannot uninstall numpy 1.21.0, RECORD file not found

pip install --ignore-installed numpy -i https://pypi.doubanio.com/simple/  --trusted-host pypi.doubanio.com

OpenCV的卡尔曼滤波的参数：

CV_WRAP KalmanFilter(int dynamParams, int measureParams, int controlParams=0, int type=CV_32F);
void init(int dynamParams, int measureParams, int controlParams=0, int type=CV_32F);

Mat statePre;           //!< predicted state (x'(k)): x(k)=A*x(k-1)+B*u(k) DP*1
Mat statePost;          //!< corrected state (x(k)): x(k)=x'(k)+K(k)*(z(k)-H*x'(k))  
Mat transitionMatrix;   //!< state transition matrix (A)  DP*DP
Mat controlMatrix;      //!< control matrix (B) (not used if there is no control)  DP*CP
Mat measurementMatrix;  //!< measurement matrix (H)  MP*DP
Mat processNoiseCov;    //!< process noise covariance matrix (Q)  DP*DP
Mat measurementNoiseCov;//!< measurement noise covariance matrix (R)  MP*MP
Mat errorCovPre;        //!< priori error estimate covariance matrix (P'(k)): P'(k)=A*P(k-1)*At + Q)*/  DP*DP
Mat gain;               //!< Kalman gain matrix (K(k)): K(k)=P'(k)*Ht*inv(H*P'(k)*Ht+R)  DP*MP
Mat errorCovPost;       //!< posteriori error estimate covariance matrix (P(k)): P(k)=(I-K(k)*H)*P'(k)  DP*DP

paddle-bfloat安装失败

https://github.com/PaddlePaddle/Paddle/issues/40559
去pypi主页下载符合环境的包→paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl到本地
将paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl名称改为paddle_bfloat-0.1.2-cp37-cp37m-linux_x86_64.whl

• 之前我的pip版本是21.0.1，直接pip install paddle_bfloat，会报上面我贴的错误
• 升级pip版本到pip==22.0.4，直接pip install paddle_bfloat，会报下面错误：

Collecting paddle_bfloat
Using cached paddle_bfloat-0.1.3.tar.gz (558 kB)
Preparing metadata (setup.py) … done
Building wheels for collected packages: paddle_bfloat
Building wheel for paddle_bfloat (setup.py) … error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [13 lines of output]
    running bdist_wheel
    running build
    running build_ext
    building 'paddle_bfloat' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    gcc -pthread -B /da2/miniconda3/envs/SWHL/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/da2/miniconda3/envs/SWHL/include/python3.7m -I/da2/miniconda3/envs/SWHL/lib/python3.7/site-packages/numpy/core/include -c bfloat16.cc -o build/temp.linux-x86_64-3.7/bfloat16.o -std=c++11
    creating build/lib.linux-x86_64-3.7
    g++ -pthread -shared -B /da2/miniconda3/envs/SWHL/compiler_compat -L/da2/miniconda3/envs/SWHL/lib -Wl,-rpath=/da2/miniconda3/envs/SWHL/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/bfloat16.o -o build/lib.linux-x86_64-3.7/paddle_bfloat.cpython-37m-x86_64-linux-gnu.so
    /da2/miniconda3/envs/SWHL/compiler_compat/ld: cannot find /usr/lib64/libmvec_nonshared.a
    /da2/miniconda3/envs/SWHL/compiler_compat/ld: cannot find /lib64/libmvec.so.1
    collect2: error: ld returned 1 exit status
    error: command 'g++' failed with exit status 1
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for paddle_bfloat
Running setup.py clean for paddle_bfloat

Failed to build paddle_bfloat
Installing collected packages: paddle_bfloat
Running setup.py install for paddle_bfloat … error
error: subprocess-exited-with-error

× Running setup.py install for paddle_bfloat did not run successfully.
│ exit code: 1
╰─> [13 lines of output]
    running install
    running build
    running build_ext
    building 'paddle_bfloat' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    gcc -pthread -B /da2/miniconda3/envs/SWHL/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/da2/miniconda3/envs/SWHL/include/python3.7m -I/da2/miniconda3/envs/SWHL/lib/python3.7/site-packages/numpy/core/include -c bfloat16.cc -o build/temp.linux-x86_64-3.7/bfloat16.o -std=c++11
    creating build/lib.linux-x86_64-3.7
    g++ -pthread -shared -B /da2/miniconda3/envs/SWHL/compiler_compat -L/da2/miniconda3/envs/SWHL/lib -Wl,-rpath=/da2/miniconda3/envs/SWHL/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/bfloat16.o -o build/lib.linux-x86_64-3.7/paddle_bfloat.cpython-37m-x86_64-linux-gnu.so
    /da2/miniconda3/envs/SWHL/compiler_compat/ld: cannot find /usr/lib64/libmvec_nonshared.a
    /da2/miniconda3/envs/SWHL/compiler_compat/ld: cannot find /lib64/libmvec.so.1
    collect2: error: ld returned 1 exit status
    error: command 'g++' failed with exit status 1
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: legacy-install-failure

× Encountered error while trying to install package.
╰─> paddle_bfloat

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

我尝试去paddle-bfloat 的pypi主页找符合我这里环境的包→paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl，将它下载到本地，直接安装

$ pip install paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl

报错如下：

ERROR: paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl is not a supported wheel on this platform.

将paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl名称改为paddle_bfloat-0.1.2-cp37-cp37m-linux_x86_64.whl

$ pip install paddle_bfloat-0.1.2-cp37-cp37m-linux_x86_64.whl
Processing ./paddle_bfloat-0.1.2-cp37-cp37m-linux_x86_64.whl
Installing collected packages: paddle-bfloat
Successfully installed paddle-bfloat-0.1.2

总算是成功解决了，感谢感谢

关于bert

Conda SSL Error
问题

更新windows或更新anconda后出现问题
Conda SSL Error: OpenSSL appears to be unavailable on this machine. OpenSSL is required to download。
按提示安装 openssl，问题也没解决

解决方法

分析：这是.dll错误导致的

转到安装anaconda-anaconda3>Library>bin的位置。搜索并复制以下dll文件
    libcrypto-1_1-x64.dll
    libssl-1_1-x64.dll

然后复制这两个文件到anaconda3>DLLs.

————————————————
问题：
Microsoft Visual C++ 14.0 is required. Get it with “Microsoft Visual C++ Build Tools
解决方法：
conda install libpython m2w64-toolchain -c msys2