问题记录——TypeError: buffer is too small for requested array

RanceGru

于 2023-05-15 16:31:23 发布

阅读量1.3k

点赞数

分类专栏：问题记录文章标签： python 深度学习神经网络

本文链接：https://blog.csdn.net/qq_45848817/article/details/130686774

版权

问题记录专栏收录该内容

2 篇文章 0 订阅

订阅专栏

问题：buffer=self.weights_file.read(param_size * 4)) TypeError: buffer is too small for requested array

在做模型量化和模型转换时遇到的一个问题，主要表现为在转换一个并不复杂的模型时，转换到一半提示：

TypeError: buffer is too small for requested array

在这里插入图片描述

查了很多地方，对此有很多原因可以造成这种报错，比如：
1、cfg文件与模型文件不匹配
2、文件名或者路径名不合规导致
3、模型训练时，数据缺失
4、onnx的操作符不完整

解决方法

我的主要是onnx的操作符不完整，就是route层的逻辑问题，比如以下两种route层的用法是不同的，在onnx中单纯的Concat操作符是无法完成两种操作的。

在这里插入图片描述

所以左边的route是分化，使用Split操作符
而右边的route是相加，使用Concat操作符

具体方法就是修改darknet2onnx的def _make_route_node代码，比如：

    def _make_route_node(self, layer_name, layer_dict):
        """If the 'layers' parameter from the DarkNet configuration is only one index, continue
        node creation at the indicated (negative) index. Otherwise, create an ONNX Concat node
        with the route properties from the DarkNet-based graph.

        Keyword arguments:
        layer_name -- the layer's name (also the corresponding key in layer_configs)
        layer_dict -- a layer parameter dictionary (one element of layer_configs)
        """
        route_node_indexes = layer_dict['layers']
        if len(route_node_indexes) == 1:
            if 'groups' in layer_dict.keys():
                # for CSPNet-kind of architecture
                assert 'group_id' in layer_dict.keys()
                groups = layer_dict['groups']
                group_id = int(layer_dict['group_id'])
                assert group_id < groups
                index = route_node_indexes[0]
                if index > 0:
                    # +1 for input node (same reason as below)
                    index += 1
                route_node_specs = self._get_previous_node_specs(target_index=index)
                assert route_node_specs.channels % groups == 0
                channels = route_node_specs.channels // groups
                outputs = [layer_name + '_%d' % i for i in range(groups)]
                outputs[group_id] = layer_name
                route_node = helper.make_node(
                    'Split',
                    axis=1,
                    split=[channels] * groups,
                    inputs=[route_node_specs.name],
                    outputs=outputs,
                    name=layer_name,
                )
                self._nodes.append(route_node)
            else:
                index = route_node_indexes[0]
                if index > 0:
                    index += 1
                route_node_specs = self._get_previous_node_specs(
                    target_index=index)
                layer_name = route_node_specs.name
                channels = route_node_specs.channels
        else:
            inputs = list()
            channels = 0
            for index in route_node_indexes:
                if index > 0:
                    # Increment by one because we count the input as a node (DarkNet
                    # does not)
                    index += 1
                route_node_specs = self._get_previous_node_specs(
                    target_index=index)
                inputs.append(route_node_specs.name)
                channels += route_node_specs.channels
            assert inputs
            assert channels > 0

            route_node = helper.make_node(
                'Concat',
                axis=1,
                inputs=inputs,
                outputs=[layer_name],
                name=layer_name,
            )
            self._nodes.append(route_node)
        return layer_name, channels