image2column的实现

一、背景

caffe的卷积实现就是 image to column，这次学习并实现一下，计划实现如下

1. N C H W 格式的python实现
2. N C H W 格式的C++实现

关于image2column的原理自行搜索，比较简单直白，不在这里说了。

注意：下面的代码实现不考虑浅拷贝和深拷贝问题，主要是想说明image2column这个算法

二、python/pytorch 实现

2. 1 最简单的情况，N=1 C=1,kernel size=3的实现

# 生成 feature map和kernel
im = torch.rand(1, 1, 16,16)
kernel_data = torch.rand(1,1,3,3)

# 利用pytorch的卷积 生成输出
conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(3,3),stride=1, padding=1, padding_mode='zeros', bias=False)
conv.weight = nn.Parameter(kernel_data)
output = conv(im)

""" image to col  """

input_shape = im.shape
kernel_shape = conv.weight.data.shape
# 预分配im2col的matrix
im_col = torch.zeros(input_shape[-1]*input_shape[-2], kernel_shape[-1]*kernel_shape[-2]) #

padded_im = nn.functional.pad(im,(1,1,1,1),'constant', 0)
padded_input_shape = padded_im.shape
# 截取每个3×3的子区域并塞入到im_col 里面
k = 0
for i in range(1, padded_input_shape[-2]-1):
    for j in range(1, padded_input_shape[-1]-1):
        im_col[k, :] = padded_im[0, 0, i-1:i+2,j-1:j+2].clone().reshape(-1)
        k += 1

# mat_col × kernel 并会reshape回原尺度。注意这里是矩阵乘法
output_mat = torch.matmul(im_col, kernel_data.view(9, 1))
output_mat_reshape = output_mat.reshape(1, 1, 16, 16)

image2column的结果output_mat_reshape, 和pytorch卷积算子的结果 output是一致的

2.2 自定义的N/C，但是kernel size还是坚持为3

# 自定义 feature map大小，当然，kernel大小还是3
out_c, n, in_c, h, w =5, 2, 3, 13, 16
im = torch.rand(n, in_c, h, w)
kernel_data = torch.rand(out_c, in_c, 3, 3)
kernel_size = kernel_data.shape[-1]
print("input feature map shape is {}, kernel shape is {}".format(im.shape, kernel_data.shape))

# 使用torch自带的卷积，屏蔽掉bias
conv = nn.Conv2d(in_channels=in_c, out_channels=out_c, kernel_size=(3,3),stride=1, padding=1, padding_mode='zeros', bias=False)
conv.weight = nn.Parameter(kernel_data)
output = conv(im) # 

print("shape of output by torch.conv is {}".format(output.shape))


# 对feature map进行手动pad
padded_im = nn.functional.pad(im,(1,1,1,1),'constant', 0)
print("shape of padded feature map is {}".format(padded_im.shape))

# 预分配im2col的matrix:
# 大小为 [n * h * w, in_c * kernel_size * kernel_size]
im_col = torch.zeros(n * w * h, in_c * kernel_size * kernel_size)

# 截取每个3×3的子区域并塞入到im2col矩阵里面
padded_input_shape = padded_im.shape
k = 0
for idx_im in range(n):
    for i in range(1, padded_input_shape[-2]-1):
        for j in range(1, padded_input_shape[-1]-1):
            im_col[k, :] = padded_im[idx_im, :, i-1:i+2,j-1:j+2].clone().reshape(-1)
            k += 1
            
# im2col和reshape后的kernel进行相乘
# reshape后的kernel大小为[kernel_size*kernel_size*in_c, out_c]
output_mat = torch.matmul(im_col, kernel_data.reshape(kernel_size*kernel_size*in_c, out_c))
print(output_mat.shape)# [n * w * h, out_c]

# 将结果reshape，这里面维度处理需要注意
output_mat_reshape = output_mat.permute(1, 0) # [out_c, n * h * w] 
output_mat_reshape = output_mat_reshape.reshape(out_c, n , h, w)  # [out_c, n,  h,w] 
output_mat_reshape = output_mat_reshape.permute(1, 0, 2,3)# [ n, out_c, h,w] 

本例中 输入是 [2, 3, 13, 16] kernel是 [5, 3, 3 ,3] 输出是[2,5,13,16]
image2column的结果output_mat_reshape, 和pytorch卷积算子的结果 output是一致的
.
.
.
.
.
.
.
.
.
.