TensorFlow优化卷积核技术演示

一 简介

在实际的卷积训练中，为了加速，常常把卷积核裁开。比如一个3*3的过滤器，可以裁成3*1和1*3两个过滤器，分别对原有的输入做卷积操作，这样可以大大提高运算速度。

原理：在浮点运算中乘法消耗的资源比较多，我们目的就是尽量减小乘法运算。

比如对一个5*2的原始图片进行一次3*3的同卷积，相当于生成的5*2像素中每一个都要经历3*3次乘法，那么一共有90次。

同样是这个图片，如果先进行一次3*1的同卷积需要30次运算，再进行一次1*3的同卷积，一共60次。

这仅仅是一个很小的数据张量，而且随着张量维度的增大，层数的增多，减少的运算会更多。那么运算量减少了，运算效果等价吗？答案是肯定的。因为有公式做保证3*1的矩阵乘上1*3的矩阵会正好生成3*3的矩阵。所以这个技巧在卷积网络中很常见。

二 实例

1 代码

import cifar10_input
import tensorflow as tf
import numpy as np
batch_size = 128
data_dir = '/tmp/cifar10_data/cifar-10-batches-bin'
print("begin")
images_train, labels_train = cifar10_input.inputs(eval_data = False,data_dir = data_dir, batch_size = batch_size)
images_test, labels_test = cifar10_input.inputs(eval_data = True, data_dir = data_dir, batch_size = batch_size)
print("begin data")
def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)
def bias_variable(shape):
  initial = tf.constant(0.1, shape=shape)
  return tf.Variable(initial)
  
def conv2d(x, W):
  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')  
                        
def avg_pool_6x6(x):
  return tf.nn.avg_pool(x, ksize=[1, 6, 6, 1],
                        strides=[1, 6, 6, 1], padding='SAME')
# tf Graph Input
x = tf.placeholder(tf.float32, [None, 24,24,3]) # cifar data image of shape 24*24*3
y = tf.placeholder(tf.float32, [None, 10]) # 0-9 数字=> 10 classes
W_conv1 = weight_variable([5, 5, 3, 64])
b_conv1 = bias_variable([64])
x_image = tf.reshape(x, [-1,24,24,3])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
#########################################################new########
W_conv21 = weight_variable([5, 1, 64, 64])
b_conv21 = bias_variable([64])
h_conv21 = tf.nn.relu(conv2d(h_pool1, W_conv21) + b_conv21)
W_conv2 = weight_variable([1, 5, 64, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_conv21, W_conv2) + b_conv2)
###########################################################old#########
#W_conv2 = weight_variable([5, 5, 64, 64])
#b_conv2 = bias_variable([64])
#h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
###################################################################
h_pool2 = max_pool_2x2(h_conv2)
W_conv3 = weight_variable([5, 5, 64, 10])
b_conv3 = bias_variable([10])
h_conv3 = tf.nn.relu(conv2d(h_pool2, W_conv3) + b_conv3)
nt_hpool3=avg_pool_6x6(h_conv3)#10
nt_hpool3_flat = tf.reshape(nt_hpool3, [-1, 10])
y_conv=tf.nn.softmax(nt_hpool3_flat)
cross_entropy = -tf.reduce_sum(y*tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
sess = tf.Session()
sess.run(tf.global_variables_initializer())
tf.train.start_queue_runners(sess=sess)
for i in range(15000):#20000
  image_batch, label_batch = sess.run([images_train, labels_train])
  label_b = np.eye(10,dtype=float)[label_batch] #one hot
  
  train_step.run(feed_dict={x:image_batch, y: label_b},session=sess)
  
  if i%200 == 0:
    train_accuracy = accuracy.eval(feed_dict={
        x:image_batch, y: label_b},session=sess)
    print( "step %d, training accuracy %g"%(i, train_accuracy))
image_batch, label_batch = sess.run([images_test, labels_test])
label_b = np.eye(10,dtype=float)[label_batch]#one hot
print ("finished！ test accuracy %g"%accuracy.eval(feed_dict={
     x:image_batch, y: label_b},session=sess))

2 运算结果

begin

begin data

step 0, training accuracy 0.101562

step 200, training accuracy 0.34375

step 400, training accuracy 0.390625

step 600, training accuracy 0.4375

step 800, training accuracy 0.429688

step 1000, training accuracy 0.4375

step 1200, training accuracy 0.453125

step 1400, training accuracy 0.46875

step 1600, training accuracy 0.507812

step 1800, training accuracy 0.460938

step 2000, training accuracy 0.539062

step 2200, training accuracy 0.46875

...

3 说明

将原来第二层的5*5的卷积操作conv2注释掉，换成两个5*1和1*5的卷积操作，代码运行后可以看到准确率没有变化，但是速度快了一些。