mmpycocotools包安装的问题：源码安装出现：“gcc: error : ../common/maksApi.c: 没有那个文件或目录“

已于 2023-08-24 10:11:25 修改
阅读量2.1k 收藏 15
点赞数 9
文章标签： python
于 2023-08-23 22:08:04 首次发布
本文链接：https://blog.csdn.net/qq_43591172/article/details/132460701
版权

mmdetection框架中的mmpycocotools包的安装问题解决

问题背景

在这里插入图片描述

在配一篇detection论文时，安装README文档进行配置，由于论文代码采用的mmdetection(2.X.X)版本中使用了mmpycocotools这个模块（也就是pycocotools外面再套了一层），我环境当中安装了mmdet 3.0.1运行起来会出现如下的错误：

(oln) jiang@jiang:~/model/object_localization_network/object_localization_network-main$ bash tools/dist_train.sh configs/oln_box/oln_box.py 1
config file is : configs/oln_box/oln_box.py
the number of gpus is : 1
the port is : 29500
Traceback (most recent call last):
  File "tools/train.py", line 15, in <module>
    from mmdet.apis import set_random_seed, train_detector
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/__init__.py", line 1, in <module>
    from .inference import (async_inference_detector, inference_detector,
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/inference.py", line 11, in <module>
    from mmdet.datasets import replace_ImageToTensor
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/__init__.py", line 2, in <module>
    from .coco import CocoDataset
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/coco.py", line 21, in <module>
    assert pycocotools.__version__ >= '12.0.2'
AttributeError: module 'pycocotools' has no attribute '__version__'

报错的源码如下（assert引起的版本校验错误）：

try:
	import pycocotools
	if not hasattr(pycocotools, '__sphinx_mock__'):	# for doc generation
		assert pycocotools.__version__>='12.0.2'
except AssertionError:
	raise AssertionError('Incompatible version of pycocotools is installed.'
									'Run pip uninstall pycocotools first. Then run pip'
									'install mmpycocotools to install open-mmlab forked'
									'pycocotools.')

解决方案：

起初mmdet拥有自己的一个mmpycocotools工具包，功能和pycocotools类似。
最新版的mmdet框架已经遗弃了mmpycocotools这个包，也是采用通用的pycocotools。（emmm，话不多说）
现有的方案有两个：

  • 采用最新版的mmdet来运行模型，无需安装mmpycocotools包
  • 按照论文的mmdet版本来进行，安装mmpycocotools包

下面来一一说明两种方案的结果。

方案1：不安装mmpycocotools包

将代码中出现pycocotools版本验证的源码进行注释，就不会出现这种问题，但是由于mmpycocotools是在pycocotools上层（外面套了一层），所以mmpycocotools有的方法pycocotools不一定有，就会存在差异。
这个时候再继续按照这个方案进行的话，就需要对代码中的大部分与coco有关的部分进行“大手术”。所以这个方法被我pass掉了。

方案2：安装mmpycocotools包

采用pip进行在线安装，命令如下：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# pip install mmpycocotools > /home/jiang/桌面/a.txt
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-37
      creating build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      running build_ext
      Compiling pycocotools/_mask.pyx because it changed.
      [1/1] Cythonizing pycocotools/_mask.pyx
      /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-a5br_mr2/mmpycocotools_0a6b2c42ae12414093ca041fe7591fa0/pycocotools/_mask.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      building 'pycocotools._mask' extension
      sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
      creating build/common
      creating build/temp.linux-x86_64-cpython-37
      creating build/temp.linux-x86_64-cpython-37/common
      creating build/temp.linux-x86_64-cpython-37/pycocotools
      build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
      gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
      gcc: error: ../common/maskApi.c: 没有那个文件或目录
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mmpycocotools
ERROR: Could not build wheels for mmpycocotools, which is required to install pyproject.toml-based projects
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#

就会出现一个 " gcc: error: …/common/maskApi.c: 没有那个文件或目录"

下载mmpycocotools源码进行源码编译安装：

执行python setup.py build出现如下错误：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
gcc: error: ../common/maskApi.c: 没有那个文件或目录
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#

这个错误跟pip在线安装一样

后来我就尝试着在mmpycocotools根目录上一层复制一份common文件夹，本以为能够编译成功，但是接着就出现了如下的错误：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
../common/maskApi.c: In function ‘rleDecode’:
../common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |    ^~~
../common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |                                                   ^
../common/maskApi.c: In function ‘rleFrPoly’:
../common/maskApi.c:171:2: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
  171 |  for (j = 0; j < k; j++) x[j] = (int)(scale * xy[j * 2 + 0] + .5); x[k] = x[0];
      |  ^~~

gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c common/maskApi.c -o build/temp.linux-x86_64-cpython-37/common/maskApi.o
common/maskApi.c: In function ‘rleDecode’:
common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |    ^~~
common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
  235 |   if (m > 2) x += (long)cnts[m - 2]; cnts[m++] = (uint)x;
      |                                      ^~~~
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c pycocotools/_mask.c -o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o
In file included from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from pycocotools/_mask.c:1111:
/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gcc -pthread -shared -B /home/jiang/anaconda3/envs/oln/compiler_compat -L/home/jiang/anaconda3/envs/oln/lib -Wl,-rpath=/home/jiang/anaconda3/envs/oln/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-37/../common/maskApi.o build/temp.linux-x86_64-cpython-37/common/maskApi.o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o -o build/lib.linux-x86_64-cpython-37/pycocotools/_mask.cpython-37m-x86_64-linux-gnu.so
/home/jiang/anaconda3/envs/oln/compiler_compat/ld: build/temp.linux-x86_64-cpython-37/common/maskApi.o: in function `rleArea':
/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/common/maskApi.c:76: multiple definition of `rleArea'; build/temp.linux-x86_64-cpython-build/temp.linux-x86_64-cpython-37/../common/maskApi.o:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/../common/maskApi.c:225: first defined here
collect2: error: ld returned 1 exit status
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#

大概意思就是c语言一个头文件编译了两次到同一个.o文件当中，在链接的时候会出现错误。

分析：
gcc命令看不懂的可以参考 这儿

gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o

gcc编译命令里面，编译的源代码（c文件）相对路径是 ‘-c …/common/maskApi.c’，代表gcc在读取命令的时候出现绝对路径映射错误。

mmpycocotools中setup.py源码如下：

import numpy as np
from setuptools import Extension, setup

# To compile and install locally run "python setup.py build_ext --inplace"
# To install library to Python site-packages run
# "python setup.py build_ext install"
# Note that the original compile flags below are GCC flags unsupported by
# the Visual C++ 2015 build tools.
# They can safely be removed.

ext_modules = [
    Extension(
        'pycocotools._mask',
        sources=['common/maskApi.c', 'pycocotools/_mask.pyx'],
        include_dirs=[np.get_include(), 'common'],
        # extra_compile_args=['-Wno-cpp', '-Wno-unused-function', '-std=c99'],
        extra_compile_args=[],
    )
]

setup(name='mmpycocotools',
      packages=['pycocotools'],
      package_dir={'pycocotools': 'pycocotools'},
      install_requires=[
          'setuptools>=18.0', 'cython>=0.27.3', 'matplotlib>=2.1.0'
      ],
      version='12.0.3',
      ext_modules=ext_modules)

试过使用pdb工具进行调试，但是不太直观，很难发现问题所在

然后我就使用pycharm对这个模块建立项目，进行debug：
发现如下：

主要的问题出在：sources=[‘common/maskApi.c’, ‘pycocotools/_mask.pyx’],的处理上，记录如下：

  1. 问题出现地方：
    在这里插入图片描述

  2. 原来的sources为：
    在这里插入图片描述

  3. 经过步骤1图中【122】【123】行函数的处理之后，得到的新值如下：
    在这里插入图片描述

  4. 中间添加了一个"…/common/maskApi.c"源码到sources当中
    在这里插入图片描述
    最后换了另外的一个conda虚拟环境进行编译就没有这个错误

在重新编译mmpycocotools源码时发现，两者调用的build_extension(self, ext)这个函数不同，编译失败找不到"…/common/maskApi.c"是由于其调用的是Cython里面的build_extension()函数，【Cython是对原生的disutils工具包二次封装了的】，而另外一个能编译通过的环境中没有安装Cython，直接调用disutils的build_extension()函数，调用位置如下：
在这里插入图片描述

问题总结

这个问题主要虚拟环境中的Cython包所导致的；
当删除Cython包时就可以正常编译
Tips：
所使用的setuptools版本是61.2.0；
mmpycocotools版本为：12.0.3（好像mmpycocotools已经停止维护了）废了老大劲，唉！！！

做一个徘徊在牛a与牛c之间
关注
  • 9
    点赞
  • 踩
  • 15
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 10
    评论
评论 10
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

做一个徘徊在牛a与牛c之间

熬夜不易，谢谢支持！

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值