【深度Debug】OpenCV自定义层Unspecified error: Can‘t create layer、Assertion failed) inputs.size() == required

最新推荐文章于 2024-10-11 21:30:37 发布

蔡不菜和他的uU们

最新推荐文章于 2024-10-11 21:30:37 发布

阅读量643

点赞数 15

文章标签： opencv 人工智能计算机视觉

本文链接：https://blog.csdn.net/csy1021/article/details/139157853

版权

文章目录

自定义层
- 测试
Debug
常用封装的结构体
调通了
- Unspecified error: Can't create layer
- Assertion failed) inputs.size() == requiredOutputs
参考

自定义层

(OpenCV自定义层官方教程)[https://docs.opencv.org/4.9.0/dc/db1/tutorial_dnn_custom_layers.html]

class MyLayer : public cv::dnn::Layer
{
public:
	MyLayer(const cv::dnn::LayerParams &params)
	{
		// 根据params初始化
	}
	static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params)
	{
		return cv::makePtr<MyLayer>(params);
	}
	virtual bool getMemoryShapes(const std::vector<std::vector<int> > &inputs,
	const int requiredOutputs,
	std::vector<std::vector<int> > &outputs,
	std::vector<std::vector<int> > &internals) const CV_OVERRIDE
	{
		// 根据输出、layer参数逻辑等 设置 输出shape
	}
	virtual void forward(cv::InputArrayOfArrays inputs,
	cv::OutputArrayOfArrays outputs,
	cv::OutputArrayOfArrays internals) CV_OVERRIDE
	{
		// layer 处理逻辑 前向处理过程
	}
	virtual void finalize(cv::InputArrayOfArrays inputs,
	cv::OutputArrayOfArrays outputs) CV_OVERRIDE
	{
		// optional
	}
private:
T layer_arg1;
T layer_arg2;
// ...
};

void test_custom_layer()
{
CV_DNN_REGISTER_LAYER_CLASS(MyLayer, MyLayer);
//cv::dnn::LayerFactory::registerLayer("MyCrop", MyCropLayer::create);
cv::dnn::Net net;

cv::dnn::LayerParams layerParams;
layerParams.set("xstart", 10);
layerParams.set("xend", 20);
layerParams.set("ystart", 10);
layerParams.set("yend", 20);
//int cropLayerId = net.addLayer("MyLayer", "MyLayer", layerParams);

int cropLayerId = net.addLayerToPrev("MyLayer", "MyLayer", layerParams);
if (cropLayerId < 0) {
    std::cerr << "Failed to add layer" << std::endl;
    return;
}
//auto layer = net.getLayer(cropLayerId);

// 创建输入和目标形状
int in_size[] = {1, 3, 80, 80};
cv::Mat input = cv::Mat::ones(4, in_size, CV_32F);

// 设置网络输入
net.setInput(input);

// 前向传播
cv::Mat output = net.forward();

// 显示输出形状
std::cout << "Output shape: " << output.size << std::endl;
}

测试

自定义层注册

两种方式【官方教程使用第一种】
CV_DNN_REGISTER_LAYER_CLASS(MyXXX, MyXXXLayer);
//cv::dnn::LayerFactory::registerLayer(“MyCrop”, MyCropLayer::create);

这里的名称需要注意，第一个是别名，第二个是类名，后续添加层使用别名，而不是使用类名

使用错误，可能引发 Unspecified error:Can’t create layer xxx of xxx 错误，报错时，注意排查这一点

自定义层构建

创建个 net，添加自定义层

cv::dnn::Net net;

cv::dnn::LayerParams layerParams;
layerParams.set("xstart", 10);
layerParams.set("xend", 20);
layerParams.set("ystart", 10);
layerParams.set("yend", 20);

//int cropLayerId = net.addLayer("MyLayer", "MyLayer", layerParams);

int cropLayerId = net.addLayerToPrev("MyLayer", "MyLayer", layerParams);
if (cropLayerId < 0) {
    std::cerr << "Failed to add layer" << std::endl;
    return;
}

==最好不要使用net.addLayer，这种还需要使用connect() 连接层与层，使用addLayerToPrev() 添加自定义层自动与前一层连接 ==

自定义层推理

// 创建输入和目标形状
int in_size[] = {1, 3, 80, 80};
cv::Mat input = cv::Mat::ones(4, in_size, CV_32F);

// 设置网络输入
net.setInput(input);

// 前向传播
cv::Mat output = net.forward();

// 显示输出形状
std::cout << "Output shape: " << output.size << std::endl;

Debug

涉及文件
layer_factory.cpp
net.cpp
net_impl.cpp

net 在创建过程中，会添加一个 _input 层
在这里插入图片描述

中间会经常涉及到 Layer 的构建与获取，【OpenCV 定义一个工厂类，会把OpenCV 定义好的层放进去，会调用工厂的静态方法createLayerInstance 来根据 type 调用工厂实现类创建 layer；】，因此下面的方法会经常调用，我不明白为什么这样设计

net.cpp —> Ptr Net::getLayer(int layerId) const

net_impl.cpp —> Ptr Net::Impl::getLayer(int layerId) const

net_impl.cpp —> Ptr getLayerInstance(LayerData& ld) const

createLayerInstance()

return ld.layerInstance

layer_factory.cpp —> Ptr LayerFactory::createLayerInstance(const String& type, LayerParams& params)

layer_factory.cpp —> LayerFactory_Impl& getLayerFactoryImpl()

在这里插入图片描述
创建好 LayerData 【opencv 好像是并不会真正的创建好这样一个模型，而是根据模型的结构 layer id 在整个过程中不断获取 layer，判断，然后在真正推理的时候才会去调用 getMemoryShapes 、forward 方法】

main —> setInput()

net.cpp —> void Net::setInput(InputArray blob, const String& name, double scalefactor, const Scalar& mean)

return impl->setInput(blob, name, scalefactor, mean);

net_impl.cpp —> void Net::Impl::setInput(InputArray blob, const String& name, double scalefactor, const Scalar& mean)
resolvePinOutputName

main—> forward()

net.cpp —> Mat Net::forward(const String& outputName)
return impl->forward(outputName);

net_impl.cpp —> Mat Net::Impl::forward(const String& outputName)

std::vector pins(1, getPinByAlias(layerName));
setUpNet(pins);
forwardToLayer(getLayerData(layerName));
return getBlob(layerName);

forward() —> SetUpNet()

net_impl.cpp —> void Net::Impl::setUpNet(const std::vector& blobsToKeep_)

clear();

allocateLayers();

完成 layer 的分配工作，会调用各个层的 getMemoryShapes，以及一些属性设定

setUpNet() —> allocateLayers()

net_impl.cpp —> void Net::Impl::allocateLayers(const std::vector& blobsToKeep_)

getLayersShapes(inputShapes, layersShapes);

net_impl.cpp —> void Net::Impl::getLayersShapes(const ShapesVec& netInputShapes,
LayersShapesMap& inOutShapes)

net_impl.cpp —> void Net::Impl::getLayerShapesRecursively(int id, LayersShapesMap& inOutShapes)

for (MapIdToLayerData::const_iterator it = layers.begin();
it != layers.end(); it++)
{
getLayerShapesRecursively(it->first, inOutShapes); 遍历获取每一个层的Shape 包括 in out internal
}

net_impl.cpp —> void Net::Impl::getLayerShapesRecursively(int id, LayersShapesMap& inOutShapes)

try
{
layerSupportInPlace = l->getMemoryShapes(is, requiredOutputs, os, ints);
}

layer_internals.hpp —> bool getMemoryShapes(const std::vector& inputs,
const int requiredOutputs,
std::vector& outputs,
std::vector& internals) const CV_OVERRIDE

CV_Assert(inputs.size() == requiredOutputs);

断言判断当前layer 输入的内容和需要其他层的输出数量上一致，如果是opencv 已经实现的层，就会调用其getMemoryShapes方法，如果是自定义层，就会进入我们自己实现的 getMemoryShapes 方法

blobManager.reset();
backendWrappers.clear();
for (auto& layer : layers)
{
auto& ld = layer.second;
ld.inputBlobsWrappers.clear();
ld.outputBlobsWrappers.clear();
ld.internalBlobsWrappers.clear();
}

然后设置 blobManager

然后分配每一层
for (MapIdToLayerData::const_iterator it = layers.begin(); it != layers.end(); it++)
{
int lid = it->first;
allocateLayer(lid, layersShapes);
}

net_impl.cpp —> void Net::Impl::allocateLayer(int lid, const LayersShapesMap& layersShapes)
{
// 操作有点复杂
// 分配完后会把各层的 flag 置为1
}

然后初始化后端
initBackend(blobsToKeep_);

forward() —> forwardToLayer

然后forwardToLayer(getLayerData(layerName));
net_impl.cpp — > void Net::Impl::forwardLayer(LayerData& ld)
{
layer->forward(inps, ld.outputBlobs, ld.internals);
}
会调用每一层的 forward 方法执行

return getBlob(layerName)
然后就完成，返回

常用封装的结构体

方便理解

LayerPin

blobsToKeep_ —> layerPin

LayerPin
{
int lid; // layerId
int oid; // outputId
}

LayerData

就是Layer，也是它内部的内容

LayerData
{
    int id;
    String name;
    String type;
    int dtype;  // Datatype of output blobs.
    LayerParams params;

    std::vector<LayerPin> inputBlobsId;
    std::set<int> inputLayersId; // 记录输入层 id
    std::set<int> requiredOutputs; // 哪些层需要本层的输出
    std::vector<LayerPin> consumers; 
    std::vector<Ptr<BackendWrapper>> outputBlobsWrappers; // 包装器
    std::vector<Ptr<BackendWrapper>> inputBlobsWrappers;
    std::vector<Ptr<BackendWrapper>> internalBlobsWrappers;

#ifdef HAVE_CUDA
    /* output ids which must be transferred to the host in the background
     * after the completion of the forward pass of the layer
     */
    std::vector<int> cudaD2HBackgroundTransfers;
#endif

    Ptr<Layer> layerInstance; 
    std::vector<Mat> outputBlobs; // 输出数据
    std::vector<Mat*> inputBlobs; // 输入数据
    std::vector<Mat> internals; // 中间值
    // Computation nodes of implemented backends (except DEFAULT).
    std::map<int, Ptr<BackendNode>> backendNodes;
    // Flag for skip layer computation for specific backend.
    bool skip; // 是否跳过本层计算

    int flag; // 是否推理过
}

LayerShapes

int Net::Impl::resolvePinOutputName(LayerData& ld, const String& outName) const{
return getLayerInstance(ld)->outputNameToIndex(outName);
}

typedef std::vector ShapesVec;
MatShape 也是个 Vector
封装了输入输出中间变量的维度
struct LayerShapes
{
ShapesVec in, out, internal;
// No guarantees that layer which support in-place computations
// will be computed in-place (input.data_ptr == output.data_ptr).
// If layer said that it could work in-place and layers after it
// no longer use input blob, we’ll set output = input.
bool supportInPlace;
LayerShapes() {supportInPlace = false;}
};