pybind11的GIL锁引入的bug

pybind11

pybind11是一个只有头文件的轻量级库，它在导出C++类型到Python的同时，也导出Python类型到C++中，其主要目的是建立现有C++代码的Python绑定。

前言

写了pybind的一个bug，导致压测场景异常core，且每次的栈不相同。

简单的demo示例

编写源码，其中py::call_guardpy::gil_scoped_release()释放GIL锁以并行提升性能

#include <pybind11/pybind11.h>

namespace py = pybind11;

// case1
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin";
    m.def("add", [](int i, int j) {
        return i + j;
        
    }, 
    py::call_guard<py::gil_scoped_release>());
}

编译安装

pip install pybind11
c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example$(python3-config --extension-suffix)

写个脚步验证它

import example
for i in range(100):
    print(example.add(10, 10))

执行结果

20
20
...

python要支持k,v形式入参

#include <pybind11/pybind11.h>

namespace py = pybind11;
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin";
    m.def("add", [](py::args args, py::kwargs kwargs) {
        int i = py::cast<int>(args[0]);
        int j = 0;
        if (kwargs.contains("j")) {
            j = py::cast<int>(kwargs["j"]);
        }
        return i + j;
        
    }, 
    py::call_guard<py::gil_scoped_release>());
}

调用方式

import example

for i in range(100):
    print(example.add(10, j=10))

此时会触发core并伴随告警

pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::inc_ref() call was triggered on a int object.
pybind11::handle::dec_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::dec_ref() call was triggered on a tuple object.
已放弃 (核心已转储)

这里已经提示代码存在问题了，释放GIL锁后，仍然存在引用计数加的问题。（但是我的arm平台执行的时候并未core，也没有编译检查）

上面提到可以用宏关掉校验，我们关掉试下：

#define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF
#include <pybind11/pybind11.h>

namespace py = pybind11;
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin";
    m.def("add", [](py::args args, py::kwargs kwargs) {
        int i = py::cast<int>(args[0]);
        int j = 0;
        if (kwargs.contains("j")) {
            j = py::cast<int>(kwargs["j"]);
        }
        return i + j;
        
    }, 
    py::call_guard<py::gil_scoped_release>());
}

然后编译，执行，一切正常。。。

20
20
...