ubuntu16安装densepose完整版--深度解决protobuf问题

本机环境：

RTX 2070

ubuntu 16.04

cuda 10.1

cudnn 7.05

在RTX2070安装densepose的话，只能安装cuda10， cuda9本人测试过程中仍存在bug

上个版本的“吐血版本”存在许多瑕疵，可能借鉴意义不大，本篇文章作为上一篇文章的补充版，应该能给大家一个很好的借鉴。

1. 首先，先安装cuda

下载deb包，按照网站说明，一步步运行即可，无须进入黑屏，方便快捷。

2. 安装caffe2

因为使用的是cuda10，而且caffe2已经被pytorch合并，因此直接安装最新cuda10版本的pytorch即可。

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch # To check if Caffe2 build was successful python2 -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure" # To check if Caffe2 GPU build was successful # This must print a number > 0 in order to use Detectron python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'

如果测试第一句有问题，可以进入python，然后import caffe2.python具体看一下是什么bug

我的是

ImportError: No module named google.protobuf.internal

解决方案

conda install -c conda-forge protobuf

这一步安装还是相当顺利，在验证这一块，我也比较顺利地通过了。

3.  安装coco API

# COCOAPI=/path/to/clone/cocoapi
git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
cd $COCOAPI/PythonAPI
# Install into global site-packages
make install
# Alternatively, if you do not have permissions or prefer
# not to install the COCO API into global site-packages
python2 setup.py install --user

4. 下载densepose代码并安装python依赖包

git clone https://github.com/facebookresearch/densepose

pip install -r $DENSEPOSE/requirements.txt

5. 编译make

cd $DENSEPOSE && make

python2 $DENSEPOSE/detectron/tests/test_spatial_narrow_as_op.py

BUG:

No handlers could be found for logger "caffe2.python.net_drawer"
net_drawer will not run correctly. Please install the correct dependencies.
[E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
Traceback (most recent call last):
  File "detectron/tests/test_spatial_narrow_as_op.py", line 80, in <module>
    c2_utils.import_detectron_ops()
  File "/home/mychocer/projects/densepose/detectron/utils/c2.py", line 33, in import_detectron_ops
    detectron_ops_lib = envu.get_detectron_ops_lib()
  File "/home/mychocer/projects/densepose/detectron/utils/env.py", line 63, in get_detectron_ops_lib
    ('Detectron ops lib not found; make sure that your Caffe2 '
AssertionError: Detectron ops lib not found; make sure that your Caffe2 version includes Detectron module

解决方法:

网上很多说把pytorch的build加到python环境变量（PYTHONPATH），网上其他教程说

/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/

加入到环境变量，但是在我的安装过程中并不奏效，而且还会带来很严重的bug。

因此我采用的方式是，

第一步：

先找到     

            libcaffe2_detectron_ops_gpu.so

sudo find / -name libcaffe2_detectron_ops_gpu.so

我的在

        /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/lib/libcaffe2_detectron_ops_gpu.so

第二步：

进入env.py文件，并且修改第54行

vim $DENSEPOSE/detectron/utils/env.py

 

prefixes = [_CMAKE_INSTALL_PREFIX, sys.prefix, sys.exec_prefix] + sys.path + ['/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/']

成功编译。

6. make ops

BUG:

CMake Error at CMakeLists.txt:8 (find_package):
  By not providing "FindCaffe2.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Caffe2", but
  CMake did not find one.

  Could not find a package configuration file provided by "Caffe2" with any
  of the following names:

    Caffe2Config.cmake
    caffe2-config.cmake

  Add the installation prefix of "Caffe2" to CMAKE_PREFIX_PATH or set
  "Caffe2_DIR" to a directory containing one of the above files.  If "Caffe2"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!

 解决方法：

查找caffe2的cmake，并加入到环境变量CMAKE_PREFIX_PATH

 

sudo find / -name Caffe2 | grep cmake

export CMAKE_PREFIX_PATH=/home/mychocer/anaconda2/envs/caffe2/share/cmake/Caffe2

BUG：

In file included from /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/utils/math.h:9:0,                                                                                        
                 from /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/utils/filler.h:8,                                                                                        
                 from /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/core/operator_schema.h:16,                                                                               
                 from /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/core/net.h:17,                                                                                           
                 from /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/core/operator.h:16,                                                                                      
                 from /home/mychocer/projects/densepose/detectron/ops/zero_even_op.h:13,
                 from /home/mychocer/projects/densepose/detectron/ops/zero_even_op.cc:9:
/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/utils/cblas.h:8:23: fatal error: mkl_cblas.h: No such file or directory                                                        
 #include <mkl_cblas.h>
                       ^

 解决方法：

不需要额外下载mkl，因为pytorch编译的时候已经安装了mkl的包，可以通过查找出来，然后加到环境变量CPATH即可

sudo find / -name mkl_cblas.h

我的文件在

   /home/mychocer/pytorch/third_party/ideep/mkl-dnn/external/mklml_lnx_2019.0.1.20180928/include/mkl_cblas.h

export CPATH=   /home/mychocer/pytorch/third_party/ideep/mkl-dnn/external/mklml_lnx_2019.0.1.20180928/include/:$CPATH

BUG：

/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/proto/caffe2.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/proto/caffe2.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/proto/caffe2.pb.h:14:2: error: #error your headers.
#error your headers.
^

解决方法：

这是protobuf版本与代码要求版本不一致导致的

第一步：

确定代码所需要版本

首先进入文件

                /home/mychocer/anaconda2/lib/python2.7/site-packages/torch/include/caffe2/proto/caffe2.pb.h

// Generated by the protocol buffer compiler.  DO NOT EDIT!
// source: caffe2/proto/caffe2.proto

#ifndef PROTOBUF_INCLUDED_caffe2_2fproto_2fcaffe2_2eproto
#define PROTOBUF_INCLUDED_caffe2_2fproto_2fcaffe2_2eproto

#include <string>

#include <google/protobuf/stubs/common.h>

#if GOOGLE_PROTOBUF_VERSION < 3006001
#error This file was generated by a newer version of protoc which is
#error incompatible with your Protocol Buffer headers.  Please update
#error your headers.
#endif
#if 3006001 < GOOGLE_PROTOBUF_MIN_PROTOC_VERSION
#error This file was generated by an older version of protoc which is
#error incompatible with your Protocol Buffer headers.  Please
#error regenerate this file with a newer version of protoc.
#endif

会看到这里所需要的protobuf版本是3006001（也就是3.6.1）

第二步：

conda安装对应版本protobuf（可以谷歌 “conda install protobuf 3.6.1”即可知道对应指令）

 conda install -c anaconda protobuf 

（可用conda search protobuf搜索看是否存在对应版本的protobuf，若存在直接conda install protobuf=3.6.1即可）

可以通过进入安装好的protobuf的common.h文件，确认版本号与上述是否对应

sudo find / -name common.h | grep google

我的地址在

         /home/mychocer/anaconda2/pkgs/libprotobuf-3.6.1-hd408876_0/include/google/protobuf/stubs/common.h

// The current version, represented as a single integer to make comparison
// easier:  major * 10^6 + minor * 10^3 + micro
#define GOOGLE_PROTOBUF_VERSION 3006001

// A suffix string for alpha, beta or rc releases. Empty for stable releases.
#define GOOGLE_PROTOBUF_VERSION_SUFFIX ""

// The minimum library version which works with the current version of the
// headers.
#define GOOGLE_PROTOBUF_MIN_LIBRARY_VERSION 3006001

// The minimum header version which works with the current version of
// the library.  This constant should only be used by protoc's C++ code
// generator.
static const int kMinHeaderVersionForLibrary = 3006001;

// The minimum protoc version which works with the current version of the
// headers.
#define GOOGLE_PROTOBUF_MIN_PROTOC_VERSION 3006001

// The minimum header version which works with the current version of
// protoc.  This constant should only be used in VerifyVersion().
static const int kMinHeaderVersionForProtoc = 3006001;

第三步：

复制protobuf的include和lib复制到anaconda的include和lib路径

### include
cp -r /home/mychocer/anaconda2/pkgs/libprotobuf-3.6.1-hd408876_0/include/google/ /home/mychocer/anaconda2/include
### lib
cp -r /home/myhcocer/anaconda2/pkgs/libprotobuf-3.6.1-hd408876_0/lib/libprotobuf* /home/mychocer/anaconda2/lib/

第四步： 

修改CMakeLists.txt

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 488ea86..b59d9bb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,11 +1,24 @@
cmake_minimum_required(VERSION 2.8.12 FATAL_ERROR)

+# set caffe2 cmake path manually
+set(Caffe2_DIR "/home/mychocer/anaconda2/lib/python2.7/site-packages/torch/share/cmake/Caffe2")
+# set cuDNN path
+#set(CUDNN_INCLUDE_DIR "/path/to/your/cudnn/include")
+#set(CUDNN_LIBRARY "/path/to/your/libcudnn/libcudnn.so")
+include_directories("/home/mychocer/anaconda2/include")
+# add static protobuf library
+add_library(libprotobuf STATIC IMPORTED)
+set(PROTOBUF_LIB "/home/mychocer/anaconda2/lib/libprotobuf.a")
+set_property(TARGET libprotobuf PROPERTY IMPORTED_LOCATION "${PROTOBUF_LIB}")
+
# Find the Caffe2 package.
# Caffe2 exports the required targets, so find_package should work for
# the standard Caffe2 installation. If you encounter problems with finding
# the Caffe2 package, make sure you have run `make install` when installing
# Caffe2 (`make install` populates your share/cmake/Caffe2).
find_package(Caffe2 REQUIRED)
+include_directories(${CAFFE2_INCLUDE_DIRS})

if (${CAFFE2_VERSION} VERSION_LESS 0.8.2)
# Pre-0.8.2 caffe2 does not have proper interface libraries set up, so we
@@ -34,19 +47,19 @@ add_library(
    caffe2_detectron_custom_ops SHARED
    ${CUSTOM_OPS_CPU_SRCS})

-target_link_libraries(caffe2_detectron_custom_ops caffe2_library)
+target_link_libraries(caffe2_detectron_custom_ops caffe2_library libprotobuf)
install(TARGETS caffe2_detectron_custom_ops DESTINATION lib)

# Install custom GPU ops lib, if gpu is present.
if (CAFFE2_USE_CUDA OR CAFFE2_FOUND_CUDA)
# Additional -I prefix is required for CMake versions before commit (< 3.7):
# https://github.com/Kitware/CMake/commit/7ded655f7ba82ea72a82d0555449f2df5ef38594
-  list(APPEND CUDA_INCLUDE_DIRS -I${CAFFE2_INCLUDE_DIRS})
+  # list(APPEND CUDA_INCLUDE_DIRS -I${CAFFE2_INCLUDE_DIRS})
CUDA_ADD_LIBRARY(
    caffe2_detectron_custom_ops_gpu SHARED
    ${CUSTOM_OPS_CPU_SRCS}
    ${CUSTOM_OPS_GPU_SRCS})

-  target_link_libraries(caffe2_detectron_custom_ops_gpu caffe2_gpu_library)
+  target_link_libraries(caffe2_detectron_custom_ops_gpu caffe2_gpu_library libprotobuf)
install(TARGETS caffe2_detectron_custom_ops_gpu DESTINATION lib)
endif()

完成！

这里安装protobuf的方式可以有很多种，可以源码安装，也可以用conda装，装完之后重复第二步到第四步即可。

 

7. 测试

python $DENSEPOSE/detectron/tests/test_zero_even_op.py

BUG：

OSError: /home/mychoer/densepose/build/libcaffe2_detectron_custom_ops_gpu.so: undefined symbol: _ZN6caffe219CPUOperatorRegistryB5cxx11Ev

解决方法：

将gcc版本从降到4.9.x即可

sudo apt-get install gcc-4.9 g++-4.9
sudo cp /usr/bin/gcc-4.9  /usr/bin/gcc
sudo cp /usr/bin/g++-4.9  /usr/bin/g++

测试成功