python extension window_使用调试Python安装在Windows上构建Python-C-Extension

If I build CPython from source on Windows I encounter problems when I want to pip install a package that contains a C-Extension. It seems like the error happens while linking the libraries.

For example when installing cython (but it also crashes with the same error on other C extension packages):

LINK : fatal error LNK1104: cannot open file 'python38.lib'

error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.23.28105\bin\HostX86\x86\link.exe' failed with exit status 1104

The reason why it cannot open the "python38.lib" is because the ".lib" file in debug mode is called "python38_d.lib".

A minimal reproducible example would be (on the command-line) based on the Quick Reference of the CPython developer guide:

git clone --branch v3.8.0 https://github.com/python/cpython.git

cd cpython

git checkout v3.8.0

.\PCbuild\build.bat -e -d

.\PCbuild\win32\python_d.exe -m ensurepip

.\PCbuild\win32\python_d.exe -m pip install pip --upgrade -vv

.\PCbuild\win32\python_d.exe -m pip install setuptools --upgrade -vv

.\PCbuild\win32\python_d.exe -m pip install cython -vv

The resulting distutils.sysconfig.get_config_vars() is:

{'BINDIR': '...\\cpython\\PCbuild\\win32',

'BINLIBDEST': ...\\cpython\\Lib',

'EXE': '.exe',

'EXT_SUFFIX': '_d.cp38-win32.pyd',

'INCLUDEPY': '...\\cpython\\include;...\\cpython\\PC',

'LIBDEST': '...\\cpython\\Lib',

'SO': '_d.cp38-win32.pyd',

'VERSION': '38',

'exec_prefix': '...\\cpython',

'prefix': '...\\cpython',

'srcdir': '...\\cpython'}

Is there something I'm missing? Is building C-Extensions on Python-debug builds on Windows simply not supported? If it is supported: how would I do it?

解决方案

Linking against pythonXY.lib is a little bit sneaky on Windows. When you look at the command line for linking, you will see that no python-library is passed to the linker, i.e. 'link.exe`. Note: This is also the case for Linux, but on Linux one doesn't have to because the needed symbols will be provided by the python-executable.

However, it is easy to check via dumpbin /dependents resulting.pyd, that there is a dependency on pythonXY.dll, also adding extra_link_args = ["/VERBOSE:LIB"] to extension-definition and triggering verbose-mode of the linker will show that the linker uses pythonXY.lib.

The sneaky part: Microsoft Compler has a convinience-pragma #pragma comment(lib, ...) to automatically trigger linking of a library, which is also used in Python-headers:

# if defined(_MSC_VER)

/* So MSVC users need not specify the .lib

file in their Makefile (other compilers are

generally taken care of by distutils.) */

# if defined(_DEBUG)

# pragma comment(lib,"python39_d.lib")

# elif defined(Py_LIMITED_API)

# pragma comment(lib,"python3.lib")

# else

# pragma comment(lib,"python39.lib")

# endif /* _DEBUG */

# endif /* _MSC_VER */

As you can see, to link against the debug version, one needs to define _DEBUG.

_DEBUG is automatically defined by distutils on Windows, if build_ext is called with options --debug, e.g.

python setup.py build_ext -i --debug

pip install --global-option build --global-option --debug XXXXX

which can be interpreted roughly as: trigger build command (which also includes build_ext-command) with option --debug prior to installing.

Another subtility when building debug C-extensions, there is more to it on Windows:

#ifdef _DEBUG

# define Py_DEBUG

#endif

Having defined Py_DEBUG macro meant incompartible ABIs until Python3.8, because it also assumed Py_TRACE_REFS which leads to different memory layout of PyObject and some additional functionality missing in the release-mode.

However, since Python3.8, one probably can get away with it by providing the missing pythonXY_d.lib/pythonYX.lib as a symlink linking to another version.