【Keras-Inception v1】CIFAR-10

系列连载目录

• 请查看博客 《Paper》 4.1 小节 【Keras】Classification in CIFAR-10 系列连载

学习借鉴

• github：BIGBALLON/cifar-10-cnn
• 知乎专栏：写给妹子的深度学习教程
• Inception v1 Caffe 代码：https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet
• Inception v1 Keras 代码：https://gist.github.com/joelouismarino/a2ede9ab3928f999575423b9887abd14
• GoogleNet网络详解与keras实现：https://blog.csdn.net/qq_25491201/article/details/78367696

参考

• 【Keras-CNN】CIFAR-10
• 本地远程访问Ubuntu16.04.3服务器上的TensorBoard
• caffe代码可视化工具

代码下载

• 链接：https://pan.baidu.com/s/1fJ1O03Ju4zuQBDdsBEO97g
  提取码：h5pu

硬件

• TITAN XP

---

1 理论基础

参考 【Inception-v1】《Going Deeper with Convolutions》

2 代码实现

[D] Why aren’t Inception-style networks successful on CIFAR-10/100?

2.1 Inception_v1

1）导入库，设置好超参数

import os  
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   
os.environ["CUDA_VISIBLE_DEVICES"]="0" 

import keras
import numpy as np
import math

from keras.datasets import cifar10
from keras.layers import Conv2D, MaxPooling2D, AveragePooling2D, ZeroPadding2D, GlobalAveragePooling2D
from keras.layers import Flatten, Dense, Dropout,BatchNormalization
from keras.models import Model
from keras.layers import Input, concatenate
from keras import optimizers, regularizers
from keras.preprocessing.image import ImageDataGenerator
from keras.initializers import he_normal
from keras.callbacks import LearningRateScheduler, TensorBoard, ModelCheckpoint

num_classes        = 10
batch_size         = 64         # 64 or 32 or other
epochs             = 300
iterations         = 782       
USE_BN=True
LRN2D_NORM = True
DROPOUT=0.4
CONCAT_AXIS=3
WEIGHT_DECAY=1e-4
DATA_FORMAT='channels_last' # Theano:'channels_first' Tensorflow:'channels_last'

log_filepath  = './inception_v1'

2）数据预处理并设置 learning schedule

def color_preprocessing(x_train,x_test):
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    mean = [125.307, 122.95, 113.865]
    std  = [62.9932, 62.0887, 66.7048]
    for i in range(3):
        x_train[:,:,:,i] = (x_train[:,:,:,i] - mean[i]) / std[i]
        x_test[:,:,:,i] = (x_test[:,:,:,i] - mean[i]) / std[i]
    return x_train, x_test

def scheduler(epoch):
    #if epoch < 150:
    #    return 0.1
    #if epoch < 225:
    #    return 0.01
    return 0.01

# load data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test  = keras.utils.to_categorical(y_test, num_classes)
x_train, x_test = color_preprocessing(x_train, x_test)

3）定义网络结构
定义LRN层，这里其实用Batch Normalization 代替了，具体LRN实现参考
https://gist.github.com/joelouismarino/a2ede9ab3928f999575423b9887abd14

Local Response Normalization 第一次出现在AlexNet论文中，原理如下：

把第 $i$ 个 channel 的某一个位置 $(x,y)$ 的像素点 $a_{x,y}^i$ 归一化，除以对应前 $n/2$ 和后 $n/2$ 共 $n+1$ 个 channels 的像素点的平方。

• $k,\alpha ,\beta ,n$ 为 hyper-parameters
  

• 其中 N 为 channels 的总和，$i,j$表示 第 $i\mathrm{th},j\mathrm{th}$个 channel，$a_{x,y}^i$ 表示 feature map（通过activation function 之后的） 中第 $i$ 个channel 的 $(x,y)$位置的值

• $\sum$ 的上下界位置限制通道数 $j$ 在 $0$ 到 $N-1$之间

def conv2D_lrn2d(x,filters,kernel_size,strides=(1,1),padding='same',data_format=DATA_FORMAT,dilation_rate=(1,1),activation='relu',use_bias=True,kernel_initializer='glorot_uniform',bias_initializer='zeros',kernel_regularizer=None,bias_regularizer=None,activity_regularizer=None,kernel_constraint=None,bias_constraint=None,lrn2d_norm=LRN2D_NORM,weight_decay=WEIGHT_DECAY):
    #l2 normalization
    if weight_decay:
        kernel_regularizer=regularizers.l2(weight_decay)
        bias_regularizer=regularizers.l2(weight_decay)
    else:
        kernel_regularizer=None
        bias_regularizer=None

    x=Conv2D(filters=filters,kernel_size=kernel_size,strides=strides,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(x)

    if lrn2d_norm:
        #batch normalization
        x=BatchNormalization()(x)

    return x

定义 inception 模块

def inception_module(x,params,concat_axis,padding='same',data_format=DATA_FORMAT,dilation_rate=(1,1),activation='relu',use_bias=True,kernel_initializer='glorot_uniform',bias_initializer='zeros',kernel_regularizer=None,bias_regularizer=None,activity_regularizer=None,kernel_constraint=None,bias_constraint=None,lrn2d_norm=LRN2D_NORM,weight_decay=None):
    (branch1,branch2,branch3,branch4)=params
    if weight_decay:
        kernel_regularizer=regularizers.l2(weight_decay)
        bias_regularizer=regularizers.l2(weight_decay)
    else:
        kernel_regularizer=None
        bias_regularizer=None
    #1x1
    pathway1=Conv2D(filters=branch1[0],kernel_size=(1,1),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(x)

    #1x1->3x3
    pathway2=Conv2D(filters=branch2[0],kernel_size=(1,1),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(x)
    pathway2=Conv2D(filters=branch2[1],kernel_size=(3,3),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(pathway2)

    #1x1->5x5
    pathway3=Conv2D(filters=branch3[0],kernel_size=(1,1),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(x)
    pathway3=Conv2D(filters=branch3[1],kernel_size=(5,5),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(pathway3)

    #3x3->1x1
    pathway4=MaxPooling2D(pool_size=(3,3),strides=1,padding=padding,data_format=DATA_FORMAT)(x)
    pathway4=Conv2D(filters=branch4[0],kernel_size=(1,1),strides=1,padding=padding,data_format=data_format,dilation_rate=dilation_rate,activation=activation,use_bias=use_bias,kernel_initializer=kernel_initializer,bias_initializer=bias_initializer,kernel_regularizer=kernel_regularizer,bias_regularizer=bias_regularizer,activity_regularizer=activity_regularizer,kernel_constraint=kernel_constraint,bias_constraint=bias_constraint)(pathway4)

    return concatenate([pathway1,pathway2,pathway3,pathway4],axis=concat_axis)

4）搭建网络

def create_model(img_input):
    x=conv2D_lrn2d(img_input,64,(7,7),2,padding='same',lrn2d_norm=False)
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)
    x=BatchNormalization()(x)

    x=conv2D_lrn2d(x,64,(1,1),1,padding='same',lrn2d_norm=False)

    x=conv2D_lrn2d(x,192,(3,3),1,padding='same',lrn2d_norm=True)
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(64,),(96,128),(16,32),(32,)],concat_axis=CONCAT_AXIS) #3a
    x=inception_module(x,params=[(128,),(128,192),(32,96),(64,)],concat_axis=CONCAT_AXIS) #3b
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(192,),(96,208),(16,48),(64,)],concat_axis=CONCAT_AXIS) #4a
    x=inception_module(x,params=[(160,),(112,224),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4b
    x=inception_module(x,params=[(128,),(128,256),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4c
    x=inception_module(x,params=[(112,),(144,288),(32,64),(64,)],concat_axis=CONCAT_AXIS) #4d
    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #4e
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #5a
    x=inception_module(x,params=[(384,),(192,384),(48,128),(128,)],concat_axis=CONCAT_AXIS) #5b
    x=AveragePooling2D(pool_size=(1,1),strides=1,padding='valid',data_format=DATA_FORMAT)(x)

    x=Flatten()(x)
    x=Dropout(DROPOUT)(x)
    x=Dense(output_dim=10,activation='linear')(x)
    x=Dense(output_dim=10,activation='softmax')(x)
    return x

5）生成模型

img_input=Input(shape=(32,32,3))
output = create_model(img_input)
model=Model(img_input,output)
model.summary()

Total params: 5,984,936
Trainable params: 5,984,424
Non-trainable params: 512

6）开始训练

# set optimizer
sgd = optimizers.SGD(lr=.1, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

# set callback
tb_cb = TensorBoard(log_dir=log_filepath, histogram_freq=0)
change_lr = LearningRateScheduler(scheduler)
cbks = [change_lr,tb_cb]

# set data augmentation
datagen = ImageDataGenerator(horizontal_flip=True,
                             width_shift_range=0.125,
                             height_shift_range=0.125,
                             fill_mode='constant',cval=0.)
datagen.fit(x_train)

# start training
model.fit_generator(datagen.flow(x_train, y_train,batch_size=batch_size),
                    steps_per_epoch=iterations,
                    epochs=epochs,
                    callbacks=cbks,
                    validation_data=(x_test, y_test))
model.save('inception_v1.h5')

7）结果分析
training accuracy 和 training loss

• accuracy
  
• loss
  
  没调 learning rate，效果还行，竟然没有vgg好，难受

---

test accuracy 和 test loss

• accuracy
  
• loss
  

…………

耐克标志，啊，过拟合了，被VGG甩了几条街，不科学！！！冷静分析，还是先解决过拟合咯

2.2 Inception_v1_n

对2.1 小节中的 inception 模块函数 def inception_module（……）形参进行如下修改，来缓解过拟合：

• kernel_initializer='glorot_uniform'替换为kernel_initializer="he_normal"
• weight_decay=None替换为weight_decay=weight_decay
• 注释掉def create_model(img_input)最后的全连接层#x=Dense(output_dim=10,activation='linear')(x)（加了和去掉影响几乎没有）
• 调整 learning rate schedule

def scheduler(epoch):
    if epoch < 100:
        return 0.01
    if epoch < 225:
        return 0.001
    return 0.0001

参数量如下：

Total params: 5,984,826
Trainable params: 5,984,314
Non-trainable params: 512

• Inception_v1
  Total params: 5,984,936

结果分析如下：
training accuracy 和 training loss

• accuracy
  
• loss
  

test accuracy 和 test loss

• accuracy
  
• loss
  

精度提升上去了，不过还是没有过90%，还是耐克，还是过拟合

2.3 Inception_v1_slim

把 Inception_v1 中 stem 结构直接替换成一个卷积，inception 结构不变，因为stem结果会把原图降到1/8的分辨率，对于 ImageNet 还行，CIFRA-10的话有些吃不消了

• 调整 learning rate schedule

def scheduler(epoch):
    if epoch < 100:
        return 0.01
    if epoch < 200:
        return 0.001
    return 0.0001

• 调整网络结构

def create_model(img_input):
    x = Conv2D(filters=192,kernel_size=(3,3),strides=(1,1),padding='same',
               kernel_initializer="he_normal",
               kernel_regularizer=regularizers.l2(weight_decay))(img_input)

    x=inception_module(x,params=[(64,),(96,128),(16,32),(32,)],concat_axis=CONCAT_AXIS) #3a
    x=inception_module(x,params=[(128,),(128,192),(32,96),(64,)],concat_axis=CONCAT_AXIS) #3b
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(192,),(96,208),(16,48),(64,)],concat_axis=CONCAT_AXIS) #4a
    x=inception_module(x,params=[(160,),(112,224),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4b
    x=inception_module(x,params=[(128,),(128,256),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4c
    x=inception_module(x,params=[(112,),(144,288),(32,64),(64,)],concat_axis=CONCAT_AXIS) #4d
    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #4e
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #5a
    x=inception_module(x,params=[(384,),(192,384),(48,128),(128,)],concat_axis=CONCAT_AXIS) #5b
    x=GlobalAveragePooling2D()(x)
    x = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
              kernel_regularizer=regularizers.l2(weight_decay))(x)
    return x

其它代码同 Inception_v1_n
参数量如下：

Total params: 5,864,762
Trainable params: 5,864,762
Non-trainable params: 0

• Inception_v1
  Total params: 5,984,936
• Inceptionv_v1_n
  Total params: 5,984,826

结果分析如下：
training accuracy 和 training loss

• accuracy
  

• loss
  

test accuracy 和 test loss

• accuracy
  
• loss
  
  精度终于上来了（92%+），过拟合缓解了，OK（hyper parameters还没优化）。
  

2.4 Auxiliary classifier

2.4.1 Inception_v1_slim_ac

ac：auxiliary classifier
如第1节理论基础所示，我们在 inception module 的 4a 和 4e 后面分别接上一个 auxiliary classifier
对 2.3 节的代码进行如下修改

• 导入库，设置超参数

from keras.layers import Lambda, Add, Activation
DROPOUT=0.3

• 调整网络结构

def create_model(img_input):
    x = Conv2D(filters=192,kernel_size=(3,3),strides=(1,1),padding='same',
               kernel_initializer="he_normal",
               kernel_regularizer=regularizers.l2(weight_decay))(img_input)

    x=inception_module(x,params=[(64,),(96,128),(16,32),(32,)],concat_axis=CONCAT_AXIS) #3a
    x=inception_module(x,params=[(128,),(128,192),(32,96),(64,)],concat_axis=CONCAT_AXIS) #3b
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(192,),(96,208),(16,48),(64,)],concat_axis=CONCAT_AXIS) #4a
    # ac1
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(aux_out)
    aux_out = Flatten()(aux_out)
    aux_out = Dense(1024,activation='relu',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out=Dropout(DROPOUT)(aux_out)
    aux_out1 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(160,),(112,224),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4b
    x=inception_module(x,params=[(128,),(128,256),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4c
    x=inception_module(x,params=[(112,),(144,288),(32,64),(64,)],concat_axis=CONCAT_AXIS) #4d
    
    # ac2
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(aux_out)
    aux_out = Flatten()(aux_out)
    aux_out = Dense(1024,activation='relu',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out=Dropout(DROPOUT)(aux_out)
    aux_out2 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #4e
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #5a
    x=inception_module(x,params=[(384,),(192,384),(48,128),(128,)],concat_axis=CONCAT_AXIS) #5b
    x=GlobalAveragePooling2D()(x)
    x = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
              kernel_regularizer=regularizers.l2(weight_decay))(x)
    
    aux_out1 = Lambda(lambda p: p * 0.3)(aux_out1)
    aux_out2 = Lambda(lambda p: p * 0.3)(aux_out2)
    output = Add()([x,aux_out1,aux_out2])
    return output

其它代码同 Inception_v1_slim
参数量如下：

Total params: 15,457,870
Trainable params: 15,457,870
Non-trainable params: 0

• Inception_v1
  Total params: 5,984,936
• Inceptionv_v1_n
  Total params: 5,984,826
• Inception_v1_slim
  Total params: 5,864,762

参数量增加了很多，主要是因为 fc 层，每一个 auxiliary classifier 有两个 fc 层
结果分析如下：
training accuracy 和 training loss

• accuracy
  

• loss
  
  inception_v1_slim 在训练集上表现的要比 inception_v1_slim_ac 好，嗯？看看测试结果再说。

test accuracy 和 test loss

• accuracy
  
• loss
  
  很好很好，至少loss更小了，精度也有稍微的提升！

2.4.2 Inception_v1_slim_ac_1

2.5.1 节中的 auxiliary classifier 有两个 fc，参数量太大，我们来简化下结构，把 ac 中第一个 fc 层换成 filter size 为 5 ， stride 为 1 的卷积，引入 batch normalization 操作，修改如下

def create_model(img_input):
    x = Conv2D(filters=192,kernel_size=(3,3),strides=(1,1),padding='same',
               kernel_initializer="he_normal",
               kernel_regularizer=regularizers.l2(weight_decay))(img_input)

    x=inception_module(x,params=[(64,),(96,128),(16,32),(32,)],concat_axis=CONCAT_AXIS) #3a
    x=inception_module(x,params=[(128,),(128,192),(32,96),(64,)],concat_axis=CONCAT_AXIS) #3b
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)
    x=inception_module(x,params=[(192,),(96,208),(16,48),(64,)],concat_axis=CONCAT_AXIS) #4a
    # ac1
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))   
    aux_out = Conv2D(1024,kernel_size=(5,5),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))       
    aux_out = Flatten()(aux_out)
    aux_out1 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(160,),(112,224),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4b
    x=inception_module(x,params=[(128,),(128,256),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4c
    x=inception_module(x,params=[(112,),(144,288),(32,64),(64,)],concat_axis=CONCAT_AXIS) #4d
    # ac2
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))   
    aux_out = Conv2D(1024,kernel_size=(5,5),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))       
    aux_out = Flatten()(aux_out)
    aux_out2 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #4e
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #5a
    x=inception_module(x,params=[(384,),(192,384),(48,128),(128,)],concat_axis=CONCAT_AXIS) #5b
    x=GlobalAveragePooling2D()(x)
    x = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
              kernel_regularizer=regularizers.l2(weight_decay))(x)
    
    aux_out1 = Lambda(lambda p: p * 0.3)(aux_out1)
    aux_out2 = Lambda(lambda p: p * 0.3)(aux_out2)
    output = Add()([x,aux_out1,aux_out2])
    return output

其它代码同 inception_v1_slim_ac
参数量如下：

Total params: 13,300,302
Trainable params: 13,295,694
Non-trainable params: 4,608

• inception_v1
  Total params: 5,984,936
• inceptionv_v1_n
  Total params: 5,984,826
• inception_v1_slim
  Total params: 5,864,762
• inception_v1_slim_ac
  Total params: 15,457,870

参数量比 inception_v1_slim_ac 降了一些，正常，因为 conv 比 fc 参数量少

2.4.3 Inception_v1_slim_ac_2

把 ac 中第一个 fc 层换成 filter size 为 3， stride 为 1 的卷积，channels 也从 1024 改为 512 ，引入 batch normalization 操作，修改如下：

def create_model(img_input):
    x = Conv2D(filters=192,kernel_size=(3,3),strides=(1,1),padding='same',
               kernel_initializer="he_normal",
               kernel_regularizer=regularizers.l2(weight_decay))(img_input)

    x=inception_module(x,params=[(64,),(96,128),(16,32),(32,)],concat_axis=CONCAT_AXIS) #3a
    x=inception_module(x,params=[(128,),(128,192),(32,96),(64,)],concat_axis=CONCAT_AXIS) #3b
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(192,),(96,208),(16,48),(64,)],concat_axis=CONCAT_AXIS) #4a
    # ac1
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))   
    aux_out = Conv2D(512,kernel_size=(3,3),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))       
    aux_out = Flatten()(aux_out)
    aux_out1 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(160,),(112,224),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4b
    x=inception_module(x,params=[(128,),(128,256),(24,64),(64,)],concat_axis=CONCAT_AXIS) #4c
    x=inception_module(x,params=[(112,),(144,288),(32,64),(64,)],concat_axis=CONCAT_AXIS) #4d
    
    # ac2
    aux_out = AveragePooling2D(pool_size=(5,5),strides=3,padding='same',data_format=DATA_FORMAT)(x)
    aux_out = Conv2D(128,kernel_size=(1,1),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))   
    aux_out = Conv2D(512,kernel_size=(3,3),strides=(1,1),padding='same',
                     kernel_initializer="he_normal",kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    aux_out = Activation('relu')(BatchNormalization(momentum=0.9, epsilon=1e-5)(aux_out))       
    aux_out = Flatten()(aux_out)
    aux_out2 = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
                     kernel_regularizer=regularizers.l2(weight_decay))(aux_out)
    
    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #4e
    x=MaxPooling2D(pool_size=(3,3),strides=2,padding='same',data_format=DATA_FORMAT)(x)

    x=inception_module(x,params=[(256,),(160,320),(32,128),(128,)],concat_axis=CONCAT_AXIS) #5a
    x=inception_module(x,params=[(384,),(192,384),(48,128),(128,)],concat_axis=CONCAT_AXIS) #5b
    x=GlobalAveragePooling2D()(x)
    x = Dense(num_classes,activation='softmax',kernel_initializer="he_normal",
              kernel_regularizer=regularizers.l2(weight_decay))(x)
    
    aux_out1 = Lambda(lambda p: p * 0.3)(aux_out1)
    aux_out2 = Lambda(lambda p: p * 0.3)(aux_out2)
    output = Add()([x,aux_out1,aux_out2])
    return output

其它代码同 inception_v1_slim_ac
参数量如下：

Total params: 7,552,590
Trainable params: 7,550,030
Non-trainable params: 2,560

• inception_v1
  Total params: 5,984,936
• inceptionv_v1_n
  Total params: 5,984,826
• inception_v1_slim
  Total params: 5,864,762
• inception_v1_slim_ac
  Total params: 15,457,870
• inception_v1_slim_ac_1
  Total params: 13,300,302

参数量进一步降低

结果分析如下：
training accuracy 和 training loss

• accuracy
  

• loss
  
  inception_v1_slim_2 和 inception_v1_slim 相仿，好于 inception_v1_slim_1

test accuracy 和 test loss

• accuracy
  
• loss
  
  inception_v1_slim_2 和 inception_v1_slim 相仿，好于 inception_v1_slim_1

2.4.4 Inception_v1_slim_ac_all

把 ac 中第一个 fc 层换成 filter size 为 3， stride 为 1 的卷积，channels 也从 1024 改为 512 ，引入 batch normalization 操作（同 Inception_v1_slim_ac_2），在所有 inception module 后加入一个如上所述的 ac 结构
其它代码同 inception_v1_slim_ac
参数量如下：

Total params: 14,269,828
Trainable params: 14,258,308
Non-trainable params: 11,520

• inception_v1
  Total params: 5,984,936
• inceptionv_v1_n
  Total params: 5,984,826
• inception_v1_slim
  Total params: 5,864,762
• inception_v1_slim_ac
  Total params: 15,457,870
• inception_v1_slim_ac_1
  Total params: 13,300,302
• inception_v1_slim_ac_2
  Total params: 7,552,590

结果分析如下：
test accuracy 和 test loss

• accuracy
  

inception_v1_slim_ac_2 效果最好

3 总结

模型大小